Oct 29 22:12:14.360061 ip-10-0-139-29 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 29 22:12:14.360073 ip-10-0-139-29 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 29 22:12:14.360081 ip-10-0-139-29 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 29 22:12:14.360397 ip-10-0-139-29 systemd[1]: Failed to start Kubernetes Kubelet. Oct 29 22:12:24.561354 ip-10-0-139-29 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 29 22:12:24.561368 ip-10-0-139-29 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0a412d3653d84b42b31d6071e1eb9fea -- Oct 29 22:14:19.896423 ip-10-0-139-29 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:14:20.296046 ip-10-0-139-29 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:20.296046 ip-10-0-139-29 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:14:20.296046 ip-10-0-139-29 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:20.296046 ip-10-0-139-29 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:20.301395 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301314 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.301395 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301395 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.301395 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301398 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301402 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301405 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301408 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301417 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301420 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301423 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301426 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301429 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301431 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301434 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301437 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301439 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301442 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301445 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301447 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301471 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301476 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301480 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301483 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.301486 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301486 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301488 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301491 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301494 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301497 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301500 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301503 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301505 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301508 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301511 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301513 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301516 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301518 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301521 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301524 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301526 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301529 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301532 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301534 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.301944 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301537 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301539 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301542 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301545 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301548 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301550 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301552 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301555 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301558 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301560 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301563 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301566 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301569 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301575 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301578 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301581 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301584 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301586 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301589 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.302411 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301592 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301594 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301597 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301601 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301603 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301607 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301609 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301612 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301615 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301617 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301621 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301626 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301630 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301633 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301636 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301638 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301641 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301644 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301646 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301649 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.302850 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301651 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301654 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301656 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301659 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301661 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301664 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301667 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301670 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301672 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301675 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301678 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301681 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301683 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301686 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301689 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301691 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301697 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301699 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.303356 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301702 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301705 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301709 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301713 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301730 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301734 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.301743 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302536 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302543 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302546 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302549 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302551 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302554 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302557 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302562 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302565 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302567 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302570 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302573 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.303793 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302576 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302579 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302582 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302585 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302587 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302590 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302593 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302595 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302598 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302601 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302604 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302606 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302609 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302612 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302614 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302617 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302620 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302622 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302625 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.304363 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302628 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302631 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302634 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302636 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302639 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302641 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302644 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302646 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302651 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302655 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302658 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302662 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302664 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302667 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302670 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302672 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302675 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302677 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302680 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.304816 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302682 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302685 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302688 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302690 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302694 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302696 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302699 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302701 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302704 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302707 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302710 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302712 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302715 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302717 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302721 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302723 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302726 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302729 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302731 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302734 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.305285 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302737 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302739 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302742 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302744 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302747 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302750 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302752 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302755 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302757 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302760 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302765 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302768 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302771 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302775 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302778 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302781 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302784 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302787 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302790 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.305761 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302793 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302795 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302798 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302800 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302803 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302805 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302808 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302811 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302813 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302816 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302819 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302821 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302824 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302827 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.302829 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303359 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303366 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303369 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303373 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303376 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.306251 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303379 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303383 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303386 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303388 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303391 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303394 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303396 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303399 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303402 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303404 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303407 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303410 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303412 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303415 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303417 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303420 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303422 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303425 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303427 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303431 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.306722 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303433 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303436 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303439 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303441 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303444 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303448 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303455 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303460 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303465 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303468 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303472 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303475 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303477 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303480 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303483 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303485 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303488 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303491 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.307186 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303493 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303496 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303498 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303501 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303504 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303506 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303508 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303511 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303514 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303517 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303519 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303521 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303524 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303527 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303529 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303532 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303535 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303537 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303540 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303543 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.307628 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303545 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303550 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303552 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303555 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303558 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303561 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303564 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303566 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303569 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303572 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303574 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303577 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303579 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303582 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303584 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303587 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303589 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303592 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303594 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.308149 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303597 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303599 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303602 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303604 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303607 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303609 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303612 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303615 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303617 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303620 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303622 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303625 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303627 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303630 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303634 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303638 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303641 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303644 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303646 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.308595 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303649 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303651 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303654 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.303657 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305506 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305518 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305524 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305529 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305534 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305538 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305542 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305547 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305550 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305553 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305557 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305560 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305564 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305567 2577 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305569 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305573 2577 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305575 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305579 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305584 2577 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305587 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:14:20.309040 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305590 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305594 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305597 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305602 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305606 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305609 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305612 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305615 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305618 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305621 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305624 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305627 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305632 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305634 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305637 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305640 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305644 2577 flags.go:64] FLAG: --enable-server="true" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305647 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305675 2577 flags.go:64] FLAG: --event-burst="100" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305679 2577 flags.go:64] FLAG: --event-qps="50" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305682 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305685 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305688 2577 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305692 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:14:20.309600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305695 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305698 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305701 2577 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305704 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305707 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305710 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305714 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305717 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305720 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305723 2577 flags.go:64] FLAG: --feature-gates="" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305727 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305731 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305734 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305737 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305740 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305743 2577 flags.go:64] FLAG: --help="false" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305746 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305750 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305752 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305756 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305759 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305763 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305766 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:14:20.310195 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305769 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305772 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305775 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305778 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305781 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305784 2577 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305787 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305790 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305793 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305796 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305799 2577 flags.go:64] FLAG: --lock-file="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305802 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305805 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305819 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305825 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305828 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305832 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305835 2577 flags.go:64] FLAG: --logging-format="text" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305838 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305841 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305844 2577 flags.go:64] FLAG: --manifest-url="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305847 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305851 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305854 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305858 2577 flags.go:64] FLAG: --max-pods="110" Oct 29 22:14:20.310741 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305861 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305864 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305867 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305870 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305873 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305876 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305879 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305886 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305889 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305892 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305895 2577 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305898 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305901 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305904 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305907 2577 flags.go:64] FLAG: --port="10250" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305910 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305912 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01214bcd31e72ff2b" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305916 2577 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305919 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305921 2577 flags.go:64] FLAG: --register-node="true" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305924 2577 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305928 2577 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305931 2577 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305934 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305937 2577 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305941 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:14:20.311340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305944 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305947 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305950 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305953 2577 flags.go:64] FLAG: --runonce="false" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305956 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305959 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305962 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305965 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305968 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305971 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305974 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305978 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305981 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305984 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305987 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305990 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305995 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.305998 2577 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306001 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306006 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306009 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306012 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306016 2577 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306019 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306022 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:14:20.311976 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306025 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306028 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306031 2577 flags.go:64] FLAG: --v="2" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306035 2577 flags.go:64] FLAG: --version="false" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306040 2577 flags.go:64] FLAG: --vmodule="" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306060 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306066 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306237 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306242 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306245 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306248 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306251 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306254 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306257 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306259 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306262 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306265 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306268 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306271 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306273 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.312602 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306278 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306281 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306285 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306288 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306290 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306293 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306296 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306298 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306301 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306304 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306306 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306309 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306312 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306315 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306317 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306320 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306323 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306326 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306329 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306332 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.313096 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306334 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306337 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306339 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306342 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306345 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306348 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306350 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306353 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306356 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306358 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306361 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306364 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306366 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306369 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306373 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306376 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306378 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306381 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306384 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306386 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.313582 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306389 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306391 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306394 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306397 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306401 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306405 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306408 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306411 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306415 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306418 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306421 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306424 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306427 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306430 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306432 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306435 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306437 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306440 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306442 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.314056 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306445 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306448 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306451 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306453 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306456 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306459 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306461 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306465 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306468 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306471 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306474 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306477 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306479 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306482 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306484 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306487 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306489 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306493 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306497 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.314509 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306500 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306503 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306510 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306513 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306516 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306519 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306522 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306524 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306527 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306529 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306532 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306535 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306537 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.306564 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306600 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.314956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306604 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306607 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306610 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306612 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306617 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306619 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306622 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306625 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306628 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306630 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306633 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306635 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306638 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306641 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306643 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306646 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306648 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306651 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306654 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306657 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.315351 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306660 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306663 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306665 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306669 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306671 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306674 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306677 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306679 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306682 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306684 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306687 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306689 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306692 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306695 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306697 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306700 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306704 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306706 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306710 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306712 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.315812 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306715 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306718 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306720 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306723 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306726 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306729 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306732 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306734 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306737 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306740 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306742 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306746 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306749 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306752 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306754 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306757 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306760 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306763 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306765 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.316314 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306771 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306775 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306778 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306782 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306784 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306787 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306790 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306792 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306795 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306798 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306801 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306804 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306806 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306809 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306811 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306814 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306816 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306819 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306821 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306824 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.316817 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306827 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306829 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306832 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306834 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306839 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306845 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306847 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306850 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306853 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306856 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306859 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306862 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306866 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306868 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306871 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306874 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306878 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.317573 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306883 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306887 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306891 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306894 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306896 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306899 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.306901 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.314102 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.314118 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.314148 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.314155 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314215 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314221 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314224 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314228 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314231 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314234 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314237 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314240 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314243 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.318282 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314245 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314248 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314251 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314253 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314256 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314259 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314261 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314264 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314269 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314272 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314275 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314278 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314281 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314284 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314287 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314289 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314292 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.319035 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314297 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314301 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314304 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314306 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314309 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314312 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314314 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314317 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314320 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314322 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314325 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314327 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314330 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314333 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314336 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314340 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314343 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314347 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314350 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.319710 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314353 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314355 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314358 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314361 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314363 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314367 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314369 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314372 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314375 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314377 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314380 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314382 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314385 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314388 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314390 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314394 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314397 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314399 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314402 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314405 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.320272 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314407 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314410 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314413 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314416 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314418 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314421 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314424 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314426 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314429 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314431 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314434 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314436 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314439 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314441 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314444 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314446 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314449 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314452 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314455 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.320745 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314457 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314460 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314465 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314467 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314470 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314473 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314475 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314478 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314481 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314484 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314488 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314491 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314494 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314497 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314500 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314503 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314505 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314508 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.321248 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314510 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.314539 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314686 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314690 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314693 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314696 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314699 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314701 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314704 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314706 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314709 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314711 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314714 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314717 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314719 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.321687 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314722 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314724 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314727 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314729 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314732 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314735 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314738 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314741 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314744 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314747 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314749 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314752 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314754 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314756 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314759 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314761 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314764 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314766 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314769 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314771 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.322114 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314774 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314777 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314779 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314782 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314785 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314787 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314790 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314792 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314795 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314798 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314800 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314803 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314806 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314808 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314811 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314814 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314816 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314819 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314821 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314824 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.322608 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314828 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314832 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314835 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314838 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314841 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314844 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314846 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314849 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314851 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314854 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314857 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314859 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314862 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314865 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314867 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314869 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314872 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314875 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314878 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.323378 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314880 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314885 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314888 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314890 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314893 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314896 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314900 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314903 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314907 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314909 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314913 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314916 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314918 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314921 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314924 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314926 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314929 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314932 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.324179 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314934 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314937 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314939 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314941 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314944 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314946 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314949 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314952 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314954 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314957 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314959 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314962 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314965 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314967 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.314970 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.314992 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:20.324948 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315024 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315028 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315032 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315035 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315038 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315040 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315043 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315046 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315048 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315051 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315054 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315057 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315061 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315065 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315068 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315071 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315074 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315076 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315079 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315082 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:20.325513 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315084 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315086 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315089 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315092 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315094 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315097 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315099 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315102 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315104 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315107 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315110 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315112 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315115 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315117 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315120 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315123 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315128 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315131 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315146 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:20.326000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315150 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315154 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315157 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315160 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315163 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315166 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315168 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315171 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315175 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315178 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315180 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315183 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315186 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315188 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315191 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315209 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315213 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315217 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315220 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315223 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:20.326493 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315225 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315228 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315231 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315233 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315236 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315239 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315242 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315244 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315247 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315251 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315253 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315256 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315259 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315261 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315264 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315266 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315268 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315271 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315273 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:20.326955 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315276 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315279 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315282 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315284 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315287 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315289 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315292 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315297 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315299 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315302 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315304 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315307 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315309 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315312 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315314 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315316 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315319 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315321 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315324 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:20.327596 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315327 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315329 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315334 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315337 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315339 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315342 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315345 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:20.315347 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.315401 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.319658 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.320766 2577 server.go:1014] "Starting client certificate rotation" Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.320864 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.320893 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.323551 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:20.328064 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.325338 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:20.337743 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.337727 2577 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:14:20.342073 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.342058 2577 log.go:25] "Validated CRI v1 image API" Oct 29 22:14:20.342905 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.342882 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:14:20.347255 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.347230 2577 fs.go:135] Filesystem UUIDs: map[0a7997db-3caf-411d-9ca1-2a363cf23032:/dev/nvme0n1p4 67bb8026-ec41-4e0c-85ac-886b2d5c78dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 29 22:14:20.347336 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.347254 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:14:20.352521 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.352392 2577 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:20.351002874 +0000 UTC m=+0.345801324 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100270 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2520a172c216540354154931caff7b SystemUUID:ec2520a1-72c2-1654-0354-154931caff7b BootID:0a412d36-53d8-4b42-b31d-6071e1eb9fea Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:75:5f:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:75:5f:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:77:f9:5e:36:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:14:20.352521 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.352515 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:14:20.352719 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.352702 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:14:20.356063 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.356032 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:14:20.356215 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.356065 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:14:20.356260 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.356217 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:14:20.356260 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.356226 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:14:20.356260 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.356236 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:20.357166 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.357152 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:20.358297 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.358286 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:14:20.362850 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.362839 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:14:20.362966 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.362957 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:14:20.363804 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.363794 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:14:20.363850 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.363807 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:14:20.363850 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.363818 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:14:20.363850 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.363828 2577 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:14:20.363850 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.363840 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:14:20.364348 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.364329 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:20.367256 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.367236 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:14:20.375880 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.375860 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:20.376824 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.376809 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:14:20.376894 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.376848 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:14:20.377046 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377035 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:14:20.377114 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377054 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:14:20.377114 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377064 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:14:20.377114 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377073 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:14:20.377114 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377083 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:14:20.377114 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377092 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:14:20.377710 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377700 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:14:20.377762 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377715 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:14:20.377762 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377728 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:14:20.377762 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377737 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:14:20.377762 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377750 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:14:20.377911 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.377765 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:14:20.378731 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.378721 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:14:20.378789 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.378734 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:14:20.382773 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.382757 2577 server.go:1263] "Started kubelet" Oct 29 22:14:20.382862 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.382821 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:14:20.382907 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.382857 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:14:20.382945 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.382924 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:14:20.383570 ip-10-0-139-29 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:14:20.384176 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.384163 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:14:20.384342 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.384327 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:14:20.389234 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.389209 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:14:20.391581 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.391564 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:20.392192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392178 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:14:20.392349 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.392326 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:20.392951 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392931 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:14:20.392951 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392951 2577 factory.go:55] Registering systemd factory Oct 29 22:14:20.393074 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392953 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:14:20.393074 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392969 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:14:20.393074 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392960 2577 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:14:20.393074 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.392953 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:14:20.393074 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.393059 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:14:20.393074 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.393067 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:14:20.393365 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.393238 2577 factory.go:153] Registering CRI-O factory Oct 29 22:14:20.393365 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.393250 2577 factory.go:223] Registration of the crio container factory successfully Oct 29 22:14:20.393365 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.393263 2577 factory.go:103] Registering Raw factory Oct 29 22:14:20.393365 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.393277 2577 manager.go:1199] Started watching for new ooms in manager Oct 29 22:14:20.394464 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.394442 2577 manager.go:319] Starting recovery of all containers Oct 29 22:14:20.404595 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.404577 2577 manager.go:324] Recovery completed Oct 29 22:14:20.406415 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.406392 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:14:20.407422 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.406961 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:14:20.413197 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.413179 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.415018 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.414997 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.415119 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415035 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.415119 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415050 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.415405 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415382 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:14:20.415485 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415452 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w764d" Oct 29 22:14:20.415697 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415684 2577 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:14:20.415765 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415697 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:14:20.415765 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415701 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.415765 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.415714 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:14:20.415898 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.414666 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-29.ec2.internal.18e31fd14a894202 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-29.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-29.ec2.internal,},FirstTimestamp:2026-10-29 22:14:20.38272461 +0000 UTC m=+0.377523059,LastTimestamp:2026-10-29 22:14:20.38272461 +0000 UTC m=+0.377523059,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-29.ec2.internal,}" Oct 29 22:14:20.416680 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.416665 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.416771 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.416697 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.416771 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.416711 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.418148 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.418123 2577 policy_none.go:50] "Start" Oct 29 22:14:20.418204 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.418153 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:14:20.418204 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.418164 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:14:20.418925 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.418911 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w764d" Oct 29 22:14:20.424014 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.423993 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 22:14:20.459237 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.459207 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 22:14:20.459475 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.459450 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:14:20.459475 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.459395 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-29.ec2.internal.18e31fd14c7601a0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-29.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-29.ec2.internal,},FirstTimestamp:2026-10-29 22:14:20.415017376 +0000 UTC m=+0.409815832,LastTimestamp:2026-10-29 22:14:20.415017376 +0000 UTC m=+0.409815832,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-29.ec2.internal,}" Oct 29 22:14:20.492751 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.492723 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:20.503659 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.503570 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-29.ec2.internal.18e31fd14c7661db default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-29.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-29.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-29.ec2.internal,},FirstTimestamp:2026-10-29 22:14:20.415042011 +0000 UTC m=+0.409840461,LastTimestamp:2026-10-29 22:14:20.415042011 +0000 UTC m=+0.409840461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-29.ec2.internal,}" Oct 29 22:14:20.551011 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.550857 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-29.ec2.internal.18e31fd14c769841 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-29.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-29.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-29.ec2.internal,},FirstTimestamp:2026-10-29 22:14:20.415055937 +0000 UTC m=+0.409854390,LastTimestamp:2026-10-29 22:14:20.415055937 +0000 UTC m=+0.409854390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-29.ec2.internal,}" Oct 29 22:14:20.551311 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.551277 2577 policy_none.go:44] "Start" Oct 29 22:14:20.580705 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.580677 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:14:20.582996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.582978 2577 manager.go:341] "Starting Device Plugin manager" Oct 29 22:14:20.583088 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.583078 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:14:20.583088 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.583088 2577 server.go:87] "Starting device plugin registration server" Oct 29 22:14:20.583314 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.583304 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:14:20.583350 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.583316 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:14:20.583447 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.583424 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:14:20.583556 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.583508 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:14:20.583556 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.583516 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:14:20.584025 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.584002 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:14:20.584106 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.584060 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:20.599905 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.599879 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:14:20.601011 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.600983 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:14:20.601011 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.601003 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:14:20.601159 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.601020 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:14:20.601159 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.601050 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 22:14:20.616996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.616974 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:20.638026 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.638002 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-29.ec2.internal\" not found" node="ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.683885 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.683855 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.684822 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.684802 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.684934 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.684838 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.684934 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.684853 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.684934 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.684887 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.701197 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.701166 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"] Oct 29 22:14:20.701286 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.701228 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.702131 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.702116 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.702244 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.702165 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.702244 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.702180 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.704226 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.704212 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.704379 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.704365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.704472 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.704397 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.704962 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.704948 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.705031 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.704948 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.705031 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.705004 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.705031 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.705016 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.705122 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.704978 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.705122 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.705055 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.705292 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.705277 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.705329 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.705301 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-29.ec2.internal\": node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:20.706982 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.706967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.707039 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.706993 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:20.707711 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.707691 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:20.707811 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.707720 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:20.707811 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.707730 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:20.737271 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.737249 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-29.ec2.internal\" not found" node="ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.740121 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.740104 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-29.ec2.internal\" not found" node="ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.750446 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.750428 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:20.795785 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.795760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.795877 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.795791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ab51ab2ff14d7b6654efe4a89f4f347-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"4ab51ab2ff14d7b6654efe4a89f4f347\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.795877 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.795813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.851348 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.851291 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:20.896611 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.896590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.896675 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.896622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.896675 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.896640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ab51ab2ff14d7b6654efe4a89f4f347-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"4ab51ab2ff14d7b6654efe4a89f4f347\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.896735 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.896698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ab51ab2ff14d7b6654efe4a89f4f347-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"4ab51ab2ff14d7b6654efe4a89f4f347\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.896735 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.896705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.896735 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:20.896705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:20.952149 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:20.952111 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:21.040239 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:21.040211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:21.043260 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:21.043244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:14:21.052557 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:21.052447 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:22.529583 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:22.529559 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:22.630053 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:22.630028 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:22.665349 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.665331 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:22.696817 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.696795 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:14:22.697167 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.696927 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:22.697167 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.696929 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:22.731101 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:22.731081 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:14:22.824036 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.823973 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:22.869236 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.869200 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:14:22.892230 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.892213 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:22.894095 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.894081 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:14:22.912493 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.912475 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:22.931609 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.931571 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:09:21 +0000 UTC" deadline="2028-05-20 05:39:07.453424822 +0000 UTC" Oct 29 22:14:22.931716 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:22.931619 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13639h24m44.521821279s" Oct 29 22:14:23.026423 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:23.026394 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab51ab2ff14d7b6654efe4a89f4f347.slice/crio-f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35 WatchSource:0}: Error finding container f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35: Status 404 returned error can't find the container with id f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35 Oct 29 22:14:23.026832 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:23.026815 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbcec17752aa0cceec127e3b277608a.slice/crio-7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00 WatchSource:0}: Error finding container 7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00: Status 404 returned error can't find the container with id 7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00 Oct 29 22:14:23.031121 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.031104 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:14:23.042259 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.042241 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:23.742681 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.742644 2577 apiserver.go:52] "Watching apiserver" Oct 29 22:14:23.769398 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.769367 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:23.785498 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.785466 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:14:23.785816 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.785796 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-68lrg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal","openshift-multus/multus-additional-cni-plugins-x4qd2","openshift-multus/network-metrics-daemon-vmv7w","openshift-ovn-kubernetes/ovnkube-node-8zr9m","openshift-cluster-node-tuning-operator/tuned-cbfz7","kube-system/konnectivity-agent-cdbjg","openshift-network-diagnostics/network-check-target-7l9b5","openshift-network-operator/iptables-alerter-7klw5","kube-system/global-pull-secret-syncer-rk9gl","kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal","openshift-dns/node-resolver-5zr4d","openshift-image-registry/node-ca-mhvkt"] Oct 29 22:14:23.788220 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.788195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.790388 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.790359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.792628 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.792606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.794954 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.794933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:23.795056 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.795021 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:23.797225 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.797205 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.799556 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.799537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.802610 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.802111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:23.804244 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.804225 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.804331 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.804272 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.804390 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.804340 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2cw6s\"" Oct 29 22:14:23.804390 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.804356 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:14:23.804390 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.804359 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:14:23.805208 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.805191 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.805302 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.805287 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:14:23.805407 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.805340 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-475gp\"" Oct 29 22:14:23.805407 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.805401 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.807512 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.807492 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:14:23.808783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.808766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:23.808871 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.808832 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:23.811511 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.811491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.814128 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.814109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.814229 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.814168 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:23.816525 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.816507 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.818936 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.818843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.837589 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.837566 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:14:23.837684 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.837657 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n8f7z" Oct 29 22:14:23.857343 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.857320 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:23.871007 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.870986 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:14:23.882455 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-kubelet-config\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.882562 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-systemd-units\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.882562 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-netns\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.882562 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-host\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.882562 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-cnibin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.882746 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-bin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.882746 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-kubelet\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.882746 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-conf-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.882830 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-multus-certs\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.882830 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-var-lib-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.882885 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-log-socket\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.882885 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-socket-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.882972 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-slash\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.882972 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-etc-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.882972 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.883060 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.883060 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.882994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/192ace5d-ce40-41f2-b494-4dd3bfae0507-konnectivity-ca\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:23.883060 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-system-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883060 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-netns\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883167 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-multus-daemon-config\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883167 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883167 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-cni-binary-copy\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883167 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-multus\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883278 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-hostroot\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883278 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-etc-kubernetes\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883278 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-kubelet\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.883278 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbw4\" (UniqueName: \"kubernetes.io/projected/690683ad-1ece-45ed-82ae-373e0a359b61-kube-api-access-lfbw4\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.883419 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c03e26-3445-48ed-9296-8678cb720276-ovn-node-metrics-cert\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.883419 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-etc-tuned\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.883419 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-sys-fs\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.883554 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-conf\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.883554 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.883554 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-script-lib\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.883720 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmftn\" (UniqueName: \"kubernetes.io/projected/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-kube-api-access-nmftn\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.883775 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-serviceca\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.883775 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/697eac89-c298-46e4-ac8c-bc825959b7bd-iptables-alerter-script\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.883775 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dwh\" (UniqueName: \"kubernetes.io/projected/697eac89-c298-46e4-ac8c-bc825959b7bd-kube-api-access-p7dwh\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-sys\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-lib-modules\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-device-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-modprobe-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-tmp\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.883910 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtl7r\" (UniqueName: \"kubernetes.io/projected/83fffeb4-7835-4126-8258-a12f509a443f-kube-api-access-jtl7r\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/192ace5d-ce40-41f2-b494-4dd3bfae0507-agent-certs\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-k8s-cni-cncf-io\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-ovn\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-cnibin\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.883992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/697eac89-c298-46e4-ac8c-bc825959b7bd-host-slash\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-tmp-dir\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnct\" (UniqueName: \"kubernetes.io/projected/9ab6261a-9f14-479b-9618-ed6672c8658d-kube-api-access-ttnct\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-systemd\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-systemd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-netd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-config\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-env-overrides\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-hosts-file\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-run\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.884317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-socket-dir-parent\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-system-cni-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5rh\" (UniqueName: \"kubernetes.io/projected/cb595d53-05aa-4575-a5e2-658dfcfbe833-kube-api-access-6r5rh\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-var-lib-kubelet\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-dbus\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-bin\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmzt\" (UniqueName: \"kubernetes.io/projected/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-kube-api-access-lmmzt\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-registration-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884321 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysconfig\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-kubernetes\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56ll\" (UniqueName: \"kubernetes.io/projected/d9e01536-6930-49aa-8507-203280a663f6-kube-api-access-r56ll\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.885049 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-node-log\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.885779 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqvh\" (UniqueName: \"kubernetes.io/projected/22c03e26-3445-48ed-9296-8678cb720276-kube-api-access-hhqvh\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.885779 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:23.885779 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.884468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-host\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.905017 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.904973 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.905150 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905094 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:14:23.905268 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905250 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:14:23.905375 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905293 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.905439 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905375 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:14:23.905439 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905376 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hznbv\"" Oct 29 22:14:23.905439 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905435 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkqw\"" Oct 29 22:14:23.905577 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905474 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.905577 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905490 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.905577 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905496 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.905577 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905540 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.905577 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905559 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.905810 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905585 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:14:23.905810 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905598 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.905810 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905641 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:14:23.905810 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905618 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm48d\"" Oct 29 22:14:23.905810 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.905543 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:14:23.918425 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918399 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:14:23.918521 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918434 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:23.918521 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918490 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbljw\"" Oct 29 22:14:23.918785 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918740 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:14:23.918874 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918860 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:14:23.918951 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918889 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kk6hd\"" Oct 29 22:14:23.918951 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918896 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxx6w\"" Oct 29 22:14:23.919055 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.918950 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:14:23.919055 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.919027 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w24c4\"" Oct 29 22:14:23.982556 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.982503 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" event={"ID":"4ab51ab2ff14d7b6654efe4a89f4f347","Type":"ContainerStarted","Data":"f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35"} Oct 29 22:14:23.983853 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.983824 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"0dbcec17752aa0cceec127e3b277608a","Type":"ContainerStarted","Data":"7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00"} Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-bin\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmzt\" (UniqueName: \"kubernetes.io/projected/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-kube-api-access-lmmzt\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-registration-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysconfig\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-kubernetes\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.985192 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985174 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-bin\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysconfig\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985284 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-registration-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56ll\" (UniqueName: \"kubernetes.io/projected/d9e01536-6930-49aa-8507-203280a663f6-kube-api-access-r56ll\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985304 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-kubernetes\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.985325 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-node-log\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.985409 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:14:24.485368554 +0000 UTC m=+3.103605987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-node-log\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqvh\" (UniqueName: \"kubernetes.io/projected/22c03e26-3445-48ed-9296-8678cb720276-kube-api-access-hhqvh\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-host\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-kubelet-config\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.985543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-systemd-units\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-netns\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985605 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-systemd-units\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-host\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-cnibin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-host\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-kubelet-config\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-bin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985725 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-bin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985753 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-netns\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-kubelet\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-conf-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985804 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-host\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-multus-certs\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-kubelet\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-var-lib-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.986261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-cnibin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-log-socket\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-socket-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-slash\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-multus-certs\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-etc-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-conf-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.985993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-var-lib-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986019 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-log-socket\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/192ace5d-ce40-41f2-b494-4dd3bfae0507-konnectivity-ca\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986064 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-slash\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-etc-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-system-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986100 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-system-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986103 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-socket-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-netns\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-multus-daemon-config\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-cni-binary-copy\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986195 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-netns\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-multus\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-hostroot\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-etc-kubernetes\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-kubelet\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbw4\" (UniqueName: \"kubernetes.io/projected/690683ad-1ece-45ed-82ae-373e0a359b61-kube-api-access-lfbw4\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-hostroot\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c03e26-3445-48ed-9296-8678cb720276-ovn-node-metrics-cert\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-etc-tuned\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-sys-fs\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-conf\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.987793 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-script-lib\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmftn\" (UniqueName: \"kubernetes.io/projected/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-kube-api-access-nmftn\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-serviceca\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986531 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/697eac89-c298-46e4-ac8c-bc825959b7bd-iptables-alerter-script\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986585 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/192ace5d-ce40-41f2-b494-4dd3bfae0507-konnectivity-ca\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986601 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-multus-daemon-config\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-sys-fs\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-etc-kubernetes\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dwh\" (UniqueName: \"kubernetes.io/projected/697eac89-c298-46e4-ac8c-bc825959b7bd-kube-api-access-p7dwh\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-cni-binary-copy\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986751 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-conf\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986788 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-kubelet\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986815 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-multus\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:23.988456 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-sys\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-lib-modules\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-device-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.986987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-modprobe-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987008 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-tmp\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtl7r\" (UniqueName: \"kubernetes.io/projected/83fffeb4-7835-4126-8258-a12f509a443f-kube-api-access-jtl7r\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.987155 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/192ace5d-ce40-41f2-b494-4dd3bfae0507-agent-certs\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-k8s-cni-cncf-io\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:23.987233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:24.487204321 +0000 UTC m=+3.105441749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-sys\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987321 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-lib-modules\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-device-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-ovn\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987359 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-k8s-cni-cncf-io\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.989177 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-cnibin\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-cnibin\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/697eac89-c298-46e4-ac8c-bc825959b7bd-host-slash\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-tmp-dir\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987494 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987852 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-serviceca\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/697eac89-c298-46e4-ac8c-bc825959b7bd-iptables-alerter-script\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnct\" (UniqueName: \"kubernetes.io/projected/9ab6261a-9f14-479b-9618-ed6672c8658d-kube-api-access-ttnct\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987940 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/697eac89-c298-46e4-ac8c-bc825959b7bd-host-slash\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-systemd\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987969 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-modprobe-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-ovn\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.987978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-systemd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988136 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-script-lib\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988149 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-systemd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-netd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.989992 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-systemd\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-config\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988240 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-netd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-env-overrides\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-hosts-file\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-run\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-socket-dir-parent\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-system-cni-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5rh\" (UniqueName: \"kubernetes.io/projected/cb595d53-05aa-4575-a5e2-658dfcfbe833-kube-api-access-6r5rh\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-var-lib-kubelet\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-dbus\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988658 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-config\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988680 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-env-overrides\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-hosts-file\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.990783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988740 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-dbus\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988782 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-var-lib-kubelet\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988792 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988823 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-system-cni-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988831 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988871 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-run\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.988882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-socket-dir-parent\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.989137 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-tmp-dir\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.989199 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:23.991501 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.991322 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-etc-tuned\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.991884 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.991569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c03e26-3445-48ed-9296-8678cb720276-ovn-node-metrics-cert\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:23.992377 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.992353 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-tmp\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:23.993292 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:23.993195 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/192ace5d-ce40-41f2-b494-4dd3bfae0507-agent-certs\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:24.006558 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.006214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmzt\" (UniqueName: \"kubernetes.io/projected/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-kube-api-access-lmmzt\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:24.006558 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.006519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqvh\" (UniqueName: \"kubernetes.io/projected/22c03e26-3445-48ed-9296-8678cb720276-kube-api-access-hhqvh\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:24.006888 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.006866 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56ll\" (UniqueName: \"kubernetes.io/projected/d9e01536-6930-49aa-8507-203280a663f6-kube-api-access-r56ll\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:24.008849 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.008829 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:24.008946 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.008855 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:24.008946 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.008870 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:24.008946 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.008934 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:24.50890477 +0000 UTC m=+3.127142207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:24.011299 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.011274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmftn\" (UniqueName: \"kubernetes.io/projected/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-kube-api-access-nmftn\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:24.012863 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.012839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbw4\" (UniqueName: \"kubernetes.io/projected/690683ad-1ece-45ed-82ae-373e0a359b61-kube-api-access-lfbw4\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:14:24.013542 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.013519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5rh\" (UniqueName: \"kubernetes.io/projected/cb595d53-05aa-4575-a5e2-658dfcfbe833-kube-api-access-6r5rh\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:24.014065 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.013950 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtl7r\" (UniqueName: \"kubernetes.io/projected/83fffeb4-7835-4126-8258-a12f509a443f-kube-api-access-jtl7r\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:24.014461 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.014400 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dwh\" (UniqueName: \"kubernetes.io/projected/697eac89-c298-46e4-ac8c-bc825959b7bd-kube-api-access-p7dwh\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:24.014831 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.014814 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnct\" (UniqueName: \"kubernetes.io/projected/9ab6261a-9f14-479b-9618-ed6672c8658d-kube-api-access-ttnct\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:24.105958 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.105919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68lrg" Oct 29 22:14:24.115902 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.115876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:14:24.118387 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.118348 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690683ad_1ece_45ed_82ae_373e0a359b61.slice/crio-abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8 WatchSource:0}: Error finding container abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8: Status 404 returned error can't find the container with id abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8 Oct 29 22:14:24.125958 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.125933 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab6261a_9f14_479b_9618_ed6672c8658d.slice/crio-d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636 WatchSource:0}: Error finding container d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636: Status 404 returned error can't find the container with id d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636 Oct 29 22:14:24.137380 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.137360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:14:24.139564 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.139540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:14:24.149202 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.149181 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb595d53_05aa_4575_a5e2_658dfcfbe833.slice/crio-7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d WatchSource:0}: Error finding container 7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d: Status 404 returned error can't find the container with id 7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d Oct 29 22:14:24.151574 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.151559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:24.151892 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.151778 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fffeb4_7835_4126_8258_a12f509a443f.slice/crio-eb56b60bca0516b14a69719b2ccea91ca8717f598141058ed9a08526d197ebe6 WatchSource:0}: Error finding container eb56b60bca0516b14a69719b2ccea91ca8717f598141058ed9a08526d197ebe6: Status 404 returned error can't find the container with id eb56b60bca0516b14a69719b2ccea91ca8717f598141058ed9a08526d197ebe6 Oct 29 22:14:24.153256 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.153042 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:14:24.161783 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.161762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:24.162277 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.162253 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192ace5d_ce40_41f2_b494_4dd3bfae0507.slice/crio-c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed WatchSource:0}: Error finding container c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed: Status 404 returned error can't find the container with id c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed Oct 29 22:14:24.163055 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.163020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:14:24.164277 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.164242 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697eac89_c298_46e4_ac8c_bc825959b7bd.slice/crio-93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d WatchSource:0}: Error finding container 93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d: Status 404 returned error can't find the container with id 93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d Oct 29 22:14:24.171028 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.171009 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:14:24.174408 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.174384 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c03e26_3445_48ed_9296_8678cb720276.slice/crio-05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d WatchSource:0}: Error finding container 05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d: Status 404 returned error can't find the container with id 05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d Oct 29 22:14:24.176078 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.176006 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3189ad37_09b3_47dc_aa9e_e384d7a0ff21.slice/crio-de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e WatchSource:0}: Error finding container de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e: Status 404 returned error can't find the container with id de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e Oct 29 22:14:24.181712 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:14:24.181670 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425b6b6b_e38b_4ceb_b473_f1c397eba5d0.slice/crio-8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239 WatchSource:0}: Error finding container 8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239: Status 404 returned error can't find the container with id 8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239 Oct 29 22:14:24.493330 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.493301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:24.493516 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.493350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:24.493516 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.493437 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:24.493516 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.493458 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:24.493516 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.493488 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:14:25.493471972 +0000 UTC m=+4.111709388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:24.493756 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.493524 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:25.493507071 +0000 UTC m=+4.111744489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:24.595258 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.594741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:24.595258 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.594930 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:24.595258 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.594952 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:24.595258 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.594965 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:24.595258 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.595014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:25.595001101 +0000 UTC m=+4.213238539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:24.841085 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.840956 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:23 +0000 UTC" deadline="2028-05-17 18:47:39.061478114 +0000 UTC" Oct 29 22:14:24.841085 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.841042 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13580h33m14.220441035s" Oct 29 22:14:24.962416 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.962287 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:24.978934 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:24.978443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:24.978934 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:24.978577 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:25.001615 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.001551 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" event={"ID":"83fffeb4-7835-4126-8258-a12f509a443f","Type":"ContainerStarted","Data":"eb56b60bca0516b14a69719b2ccea91ca8717f598141058ed9a08526d197ebe6"} Oct 29 22:14:25.014301 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.012516 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerStarted","Data":"7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d"} Oct 29 22:14:25.023018 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.022933 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" event={"ID":"9ab6261a-9f14-479b-9618-ed6672c8658d","Type":"ContainerStarted","Data":"d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636"} Oct 29 22:14:25.041895 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.041768 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d"} Oct 29 22:14:25.043685 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.043618 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7klw5" event={"ID":"697eac89-c298-46e4-ac8c-bc825959b7bd","Type":"ContainerStarted","Data":"93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d"} Oct 29 22:14:25.049047 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.048973 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68lrg" event={"ID":"690683ad-1ece-45ed-82ae-373e0a359b61","Type":"ContainerStarted","Data":"abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8"} Oct 29 22:14:25.055668 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.054947 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" event={"ID":"4ab51ab2ff14d7b6654efe4a89f4f347","Type":"ContainerStarted","Data":"468941a50a2add7ef0bdec81215abc3ac5474308bf56f5dd0f074208d334ddab"} Oct 29 22:14:25.065664 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.065085 2577 generic.go:356] "Generic (PLEG): container finished" podID="0dbcec17752aa0cceec127e3b277608a" containerID="308b20b95a7edd73c9162a5b02fd1e7f6c49e781e488d1ed35f825b091a01392" exitCode=0 Oct 29 22:14:25.065664 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.065164 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"0dbcec17752aa0cceec127e3b277608a","Type":"ContainerDied","Data":"308b20b95a7edd73c9162a5b02fd1e7f6c49e781e488d1ed35f825b091a01392"} Oct 29 22:14:25.078205 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.078181 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mhvkt" event={"ID":"425b6b6b-e38b-4ceb-b473-f1c397eba5d0","Type":"ContainerStarted","Data":"8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239"} Oct 29 22:14:25.099459 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.099426 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zr4d" event={"ID":"3189ad37-09b3-47dc-aa9e-e384d7a0ff21","Type":"ContainerStarted","Data":"de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e"} Oct 29 22:14:25.121751 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.121726 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdbjg" event={"ID":"192ace5d-ce40-41f2-b494-4dd3bfae0507","Type":"ContainerStarted","Data":"c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed"} Oct 29 22:14:25.135714 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.133241 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" podStartSLOduration=3.133222579 podStartE2EDuration="3.133222579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:25.094390413 +0000 UTC m=+3.712627853" watchObservedRunningTime="2026-10-29 22:14:25.133222579 +0000 UTC m=+3.751460017" Oct 29 22:14:25.501754 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.501614 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:25.501754 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.501716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:25.502015 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.501864 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:25.502015 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.501924 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:14:27.50190726 +0000 UTC m=+6.120144689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:25.502324 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.502306 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:25.502397 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.502360 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:27.50234622 +0000 UTC m=+6.120583650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:25.602102 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.602062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:25.602281 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.602252 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:25.602281 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.602271 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:25.602394 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.602285 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:25.602394 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.602342 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:27.602325998 +0000 UTC m=+6.220563437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:25.841944 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.841833 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:23 +0000 UTC" deadline="2028-05-23 21:43:26.124301491 +0000 UTC" Oct 29 22:14:25.841944 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.841898 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13727h29m0.282407526s" Oct 29 22:14:25.980009 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.978819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:25.980009 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.978954 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:25.980009 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:25.979394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:25.980009 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:25.979489 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:26.153215 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:26.152600 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"0dbcec17752aa0cceec127e3b277608a","Type":"ContainerStarted","Data":"d4dd3ff1e41cf2fd8127ddaa2e893529131248deb37b9f1072afc89657113ed0"} Oct 29 22:14:26.978419 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:26.978387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:26.978806 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:26.978528 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:27.158497 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.158414 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="9e9b9c361a270a7d0b3da8e6e956d4dd5cd7c4de1ccd1b93fc07ab571a9546f9" exitCode=0 Oct 29 22:14:27.159391 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.159335 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerDied","Data":"9e9b9c361a270a7d0b3da8e6e956d4dd5cd7c4de1ccd1b93fc07ab571a9546f9"} Oct 29 22:14:27.231600 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.231058 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" podStartSLOduration=5.231038853 podStartE2EDuration="5.231038853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:26.193568442 +0000 UTC m=+4.811805880" watchObservedRunningTime="2026-10-29 22:14:27.231038853 +0000 UTC m=+5.849276296" Oct 29 22:14:27.517489 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.517410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:27.517644 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.517489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:27.517644 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.517612 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:27.517759 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.517674 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:31.51765728 +0000 UTC m=+10.135894697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:27.518088 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.518068 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:27.518187 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.518124 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:14:31.518110054 +0000 UTC m=+10.136347472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:27.618522 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.618490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:27.618683 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.618647 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:27.618683 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.618669 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:27.618683 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.618682 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:27.618866 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.618748 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:31.618730289 +0000 UTC m=+10.236967719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:27.980724 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.980675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:27.981159 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.980822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:27.981159 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:27.980885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:27.981159 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:27.980950 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:28.173712 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:28.173618 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdbjg" event={"ID":"192ace5d-ce40-41f2-b494-4dd3bfae0507","Type":"ContainerStarted","Data":"9b8a1f5ff78636f7eb6b2bff12eedc516a0123dd520f43e642990f3c3f552796"} Oct 29 22:14:28.214148 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:28.213822 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cdbjg" podStartSLOduration=2.511791335 podStartE2EDuration="6.213804417s" totalImagesPullingTime="3.702013082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.212856775 +0000 UTC m=+6.831094214" watchObservedRunningTime="2026-10-29 22:14:28.213804417 +0000 UTC m=+6.832041854" Oct 29 22:14:28.978479 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:28.978446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:28.978635 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:28.978575 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:29.981194 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:29.981107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:29.981713 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:29.981258 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:29.981713 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:29.981640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:29.981937 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:29.981864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:30.179080 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:30.179045 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mhvkt" event={"ID":"425b6b6b-e38b-4ceb-b473-f1c397eba5d0","Type":"ContainerStarted","Data":"e5ad62cd6d5b655e15801d6e3d9e86538d10452a5f01813fdb346f3ea12ce3ec"} Oct 29 22:14:30.181280 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:30.181254 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" event={"ID":"9ab6261a-9f14-479b-9618-ed6672c8658d","Type":"ContainerStarted","Data":"95903fc4b96757178ab029c6476b95a7e2f69fbb5a0c6ea33bbb4199be4421a7"} Oct 29 22:14:30.221513 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:30.221485 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:30.978751 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:30.978225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:30.978751 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:30.978371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:31.221414 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.221385 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:14:31.557251 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.557159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:31.557401 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.557255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:31.557458 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.557409 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:31.557507 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.557476 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:14:39.557456882 +0000 UTC m=+18.175694320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:31.557577 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.557542 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:31.557577 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.557575 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:39.557564034 +0000 UTC m=+18.175801465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:31.657929 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.657889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:31.658572 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.658106 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:31.658572 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.658135 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:31.658572 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.658151 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:31.658572 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.658211 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:39.658193822 +0000 UTC m=+18.276431261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:31.966146 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.965807 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:31.221406575Z","UUID":"96c10dc7-8bd2-4625-a4ca-e1a5392097c9","Handler":null,"Name":"","Endpoint":""} Oct 29 22:14:31.967996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.967971 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:14:31.967996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.968002 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:14:31.987475 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.984812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:31.987475 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.985245 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:31.987475 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:31.986435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:31.987475 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:31.986885 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:32.189396 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.189363 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68lrg" event={"ID":"690683ad-1ece-45ed-82ae-373e0a359b61","Type":"ContainerStarted","Data":"eb44230bd14bbf307d95d5e96b4eab49a3a4e069dbf11270c5dff2004331b21e"} Oct 29 22:14:32.196104 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.196070 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="05fece8901432a51753f49e6cdbbd4122f6181e46281d41db5e45a7a5b372025" exitCode=0 Oct 29 22:14:32.196255 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.196158 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerDied","Data":"05fece8901432a51753f49e6cdbbd4122f6181e46281d41db5e45a7a5b372025"} Oct 29 22:14:32.200466 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.200433 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" event={"ID":"9ab6261a-9f14-479b-9618-ed6672c8658d","Type":"ContainerStarted","Data":"b5b5a5f784419e637fd9eff54f00ed4e5bfa45a415a3c6fd969526aa9dde1f59"} Oct 29 22:14:32.229818 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.229761 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mhvkt" podStartSLOduration=5.286359493 podStartE2EDuration="10.229742809s" totalImagesPullingTime="4.943383316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:30.219228821 +0000 UTC m=+8.837466258" watchObservedRunningTime="2026-10-29 22:14:32.229742809 +0000 UTC m=+10.847980247" Oct 29 22:14:32.230284 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.229871 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-68lrg" podStartSLOduration=2.619560585 podStartE2EDuration="10.229860713s" totalImagesPullingTime="7.610300128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:32.229454639 +0000 UTC m=+10.847692079" watchObservedRunningTime="2026-10-29 22:14:32.229860713 +0000 UTC m=+10.848098151" Oct 29 22:14:32.305837 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.305807 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:32.306740 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.306510 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:32.978829 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:32.978584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:32.979000 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:32.978918 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:33.204539 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.204509 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zr4d" event={"ID":"3189ad37-09b3-47dc-aa9e-e384d7a0ff21","Type":"ContainerStarted","Data":"ea26957e40692dee656906cde91b0e82ff470f896121d198d110f707bc033566"} Oct 29 22:14:33.207334 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.207300 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="23fa072687876d1ef4bd0a95a1ab4988a65c253600301a65764e9e9a3b2667c7" exitCode=0 Oct 29 22:14:33.207449 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.207355 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerDied","Data":"23fa072687876d1ef4bd0a95a1ab4988a65c253600301a65764e9e9a3b2667c7"} Oct 29 22:14:33.210337 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.210317 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" event={"ID":"9ab6261a-9f14-479b-9618-ed6672c8658d","Type":"ContainerStarted","Data":"c695538af4ec67f881c0e09629016fe91939b2f186932818366cff66a23c2414"} Oct 29 22:14:33.213017 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.212995 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7klw5" event={"ID":"697eac89-c298-46e4-ac8c-bc825959b7bd","Type":"ContainerStarted","Data":"3ebbeb4704e9ef53f539126341a4b6c2a1bf2f11b52400ecf19593643d4c63da"} Oct 29 22:14:33.213762 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.213747 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:14:33.243864 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.243823 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5zr4d" podStartSLOduration=3.262298856 podStartE2EDuration="11.243807609s" totalImagesPullingTime="7.981508753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:33.24333014 +0000 UTC m=+11.861567578" watchObservedRunningTime="2026-10-29 22:14:33.243807609 +0000 UTC m=+11.862045046" Oct 29 22:14:33.285734 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.283870 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" podStartSLOduration=3.472049126 podStartE2EDuration="11.283852483s" totalImagesPullingTime="7.811803357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:33.283519518 +0000 UTC m=+11.901756955" watchObservedRunningTime="2026-10-29 22:14:33.283852483 +0000 UTC m=+11.902089926" Oct 29 22:14:33.405366 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.405309 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7klw5" podStartSLOduration=3.900390082 podStartE2EDuration="11.40529403s" totalImagesPullingTime="7.504903948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:33.405029091 +0000 UTC m=+12.023266529" watchObservedRunningTime="2026-10-29 22:14:33.40529403 +0000 UTC m=+12.023531467" Oct 29 22:14:33.559512 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.558747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdbjg_192ace5d-ce40-41f2-b494-4dd3bfae0507/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:33.744166 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.744128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdbjg_192ace5d-ce40-41f2-b494-4dd3bfae0507/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:33.978205 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.978136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:33.978343 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:33.978253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:33.978708 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:33.978683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:33.978853 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:33.978829 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:34.217139 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:34.216942 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" event={"ID":"83fffeb4-7835-4126-8258-a12f509a443f","Type":"ContainerStarted","Data":"aed2859b42a04c1e3d453e0bc82dd723cb8ea268631ec6e41776ac6365c1d2a0"} Oct 29 22:14:34.219903 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:34.219874 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="2c10c6719862f241875148a8b61b94ecb808b231fef1ad85f5e61f9313fe1a97" exitCode=0 Oct 29 22:14:34.220006 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:34.219976 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerDied","Data":"2c10c6719862f241875148a8b61b94ecb808b231fef1ad85f5e61f9313fe1a97"} Oct 29 22:14:34.260469 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:34.260365 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" podStartSLOduration=2.328948307 podStartE2EDuration="12.260346715s" totalImagesPullingTime="9.931398408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.259976372 +0000 UTC m=+12.878213809" watchObservedRunningTime="2026-10-29 22:14:34.260346715 +0000 UTC m=+12.878584156" Oct 29 22:14:34.977855 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:34.977822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:34.978024 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:34.977956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:35.978477 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:35.978439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:35.979065 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:35.978564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:35.979065 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:35.978629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:35.979065 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:35.978758 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:36.228549 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:36.228310 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerStarted","Data":"c0bf80f94d327595101f84973d222bbf06f041f4de1c7764e07139533284d228"} Oct 29 22:14:36.978207 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:36.978179 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:36.978363 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:36.978291 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:37.233546 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:37.233439 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="c0bf80f94d327595101f84973d222bbf06f041f4de1c7764e07139533284d228" exitCode=0 Oct 29 22:14:37.233546 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:37.233500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerDied","Data":"c0bf80f94d327595101f84973d222bbf06f041f4de1c7764e07139533284d228"} Oct 29 22:14:37.978835 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:37.978795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:37.978835 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:37.978844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:37.979067 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:37.978961 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:37.979122 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:37.979099 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:38.238853 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:38.238777 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="99c0e2fc879551ce967361e3c6727cae043d837332fc5ca9625904e74c50593d" exitCode=0 Oct 29 22:14:38.238853 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:38.238819 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerDied","Data":"99c0e2fc879551ce967361e3c6727cae043d837332fc5ca9625904e74c50593d"} Oct 29 22:14:38.977911 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:38.977879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:38.978093 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:38.977977 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:39.244325 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.244297 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" event={"ID":"cb595d53-05aa-4575-a5e2-658dfcfbe833","Type":"ContainerStarted","Data":"04049a0ad46a8ac0e37d772383816f39fd1e395b8eb82cf1b0402295e7c3028e"} Oct 29 22:14:39.246397 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.246376 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"1f957fb0cb1126aa33ad55e6cc8bca38521d6609fbeaf1516e4dd6cb831fe2a7"} Oct 29 22:14:39.246472 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.246432 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"5b9c689195f332c0e37c4e127429964fd91ed8c062f2c2be89a8f0e806f78545"} Oct 29 22:14:39.246472 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.246463 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"9a0224d2a2fa3f6025e0a6b400dd9a0a0aef8420f16dcd5defe1b07920ebad42"} Oct 29 22:14:39.303659 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.303610 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x4qd2" podStartSLOduration=7.008602161 podStartE2EDuration="17.303596353s" totalImagesPullingTime="10.294994192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:39.300661339 +0000 UTC m=+17.918898778" watchObservedRunningTime="2026-10-29 22:14:39.303596353 +0000 UTC m=+17.921833791" Oct 29 22:14:39.620948 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.620925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:39.621092 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.620966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:39.621092 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.621058 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:39.621092 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.621059 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:39.621181 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.621104 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:14:55.621092839 +0000 UTC m=+34.239330254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:39.621181 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.621118 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:55.621111394 +0000 UTC m=+34.239348809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:39.721340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.721310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:39.721482 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.721459 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:39.721482 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.721478 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:39.721546 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.721487 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:39.721546 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.721538 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:55.721521425 +0000 UTC m=+34.339758852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:39.980990 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.980920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:39.980990 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:39.980948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:39.981131 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.981021 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:39.981131 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:39.981077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:40.251466 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:40.251392 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"f0d12a572d54c98af60ef0ad687667c9c25d3a27b59883b4d21ca2c793522166"} Oct 29 22:14:40.251466 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:40.251459 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"e7e4b57a2a46f10058d25559fce3399013e154e24ecb8a47a2c1b297665d23ec"} Oct 29 22:14:40.251845 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:40.251488 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"9302d4365d7c2c4a5543c9afa09038fa48bec37ca040fa97a4987152891ce6b7"} Oct 29 22:14:40.978494 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:40.978467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:40.978684 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:40.978555 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:41.980935 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:41.980910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:41.981283 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:41.981009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:41.981283 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:41.981080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:41.981283 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:41.981156 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:42.257888 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:42.257816 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"0b69c22ecbbde039ac9968b0068d21183464fed134661bfd405e30f0c40cf5c8"} Oct 29 22:14:42.978229 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:42.978199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:42.978379 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:42.978290 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:43.980506 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:43.980477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:43.981400 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:43.980630 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:43.981400 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:43.980666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:43.981400 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:43.980748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:44.265558 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:44.265481 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" event={"ID":"22c03e26-3445-48ed-9296-8678cb720276","Type":"ContainerStarted","Data":"6d7e136b1ea163e217d9eaaffba0589b36c9f3cd037d407d0bc9bdba4e6b8221"} Oct 29 22:14:44.265762 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:44.265746 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:44.280454 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:44.280429 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:44.354566 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:44.354520 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" podStartSLOduration=7.462963446 podStartE2EDuration="22.354508799s" totalImagesPullingTime="14.891545353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:44.340595981 +0000 UTC m=+22.958833418" watchObservedRunningTime="2026-10-29 22:14:44.354508799 +0000 UTC m=+22.972746235" Oct 29 22:14:44.524107 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:44.524041 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zr4d_3189ad37-09b3-47dc-aa9e-e384d7a0ff21/dns-node-resolver/0.log" Oct 29 22:14:44.977882 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:44.977852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:44.978046 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:44.977950 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:45.267445 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:45.267380 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:45.267836 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:45.267819 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:45.282065 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:45.282045 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:45.723648 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:45.723619 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mhvkt_425b6b6b-e38b-4ceb-b473-f1c397eba5d0/node-ca/0.log" Oct 29 22:14:45.980770 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:45.980690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:45.980915 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:45.980826 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:45.980915 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:45.980866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:45.980994 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:45.980978 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:46.268919 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:46.268851 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:46.978114 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:46.978080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:46.978264 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:46.978188 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:47.270980 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:47.270912 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:47.977909 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:47.977882 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:47.978075 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:47.977979 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:47.978075 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:47.978034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:47.978156 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:47.978113 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:48.978552 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:48.978530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:48.978971 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:48.978611 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:49.980596 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:49.980569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:49.980596 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:49.980590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:49.981099 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:49.980671 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:49.981099 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:49.980794 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:50.353364 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:50.353330 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:14:50.353543 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:50.353512 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:50.363588 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:50.363558 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" podUID="22c03e26-3445-48ed-9296-8678cb720276" containerName="ovnkube-controller" probeResult="failure" output="" Oct 29 22:14:50.371551 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:50.371525 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" podUID="22c03e26-3445-48ed-9296-8678cb720276" containerName="ovnkube-controller" probeResult="failure" output="" Oct 29 22:14:50.978643 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:50.978620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:50.978810 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:50.978727 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:51.981504 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:51.981476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:51.981915 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:51.981617 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:51.981915 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:51.981665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:51.981915 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:51.981786 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:52.977995 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:52.977956 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:52.978241 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:52.978074 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:53.977984 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:53.977954 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:53.978415 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:53.978055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:53.978415 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:53.978113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:53.978415 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:53.978211 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:54.978180 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:54.978140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:54.978666 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:54.978251 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:55.638534 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:55.638496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:55.638731 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:55.638553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:55.638731 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.638642 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:55.638731 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.638647 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:55.638731 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.638713 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs podName:d9e01536-6930-49aa-8507-203280a663f6 nodeName:}" failed. No retries permitted until 2026-10-29 22:15:27.638682242 +0000 UTC m=+66.256919657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs") pod "network-metrics-daemon-vmv7w" (UID: "d9e01536-6930-49aa-8507-203280a663f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:55.638731 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.638727 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret podName:2768c741-ba6f-4f8f-b20a-ac431cd8b83f nodeName:}" failed. No retries permitted until 2026-10-29 22:15:27.638720871 +0000 UTC m=+66.256958286 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret") pod "global-pull-secret-syncer-rk9gl" (UID: "2768c741-ba6f-4f8f-b20a-ac431cd8b83f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:55.739175 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:55.739136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:55.739344 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.739293 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:55.739344 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.739311 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:55.739344 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.739320 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lt97c for pod openshift-network-diagnostics/network-check-target-7l9b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:55.739447 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.739375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c podName:12e17b5f-8baa-4708-8bef-2660e45d479c nodeName:}" failed. No retries permitted until 2026-10-29 22:15:27.739359292 +0000 UTC m=+66.357596724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt97c" (UniqueName: "kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c") pod "network-check-target-7l9b5" (UID: "12e17b5f-8baa-4708-8bef-2660e45d479c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:55.978142 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:55.978052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:55.978317 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.978181 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:55.978317 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:55.978224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:55.978730 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:55.978321 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:56.978532 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:56.978491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:56.979048 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:56.978621 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:57.978805 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:57.978774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:57.979191 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:57.978941 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:14:57.979191 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:57.978995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:57.979191 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:57.979123 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:58.978789 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:58.978752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:14:58.978967 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:58.978859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:14:59.978475 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:59.978440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:14:59.978673 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:59.978580 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:14:59.978673 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:14:59.978620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:14:59.978818 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:14:59.978719 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:15:00.977788 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:00.977760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:00.978166 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:00.977842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:15:01.981313 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:01.981284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:01.981772 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:01.981425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:01.981772 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:01.981445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:15:01.981772 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:01.981538 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:15:02.689969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:02.689468 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rk9gl"] Oct 29 22:15:02.689969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:02.689634 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:02.689969 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:02.689763 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:15:02.692805 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:02.692777 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vmv7w"] Oct 29 22:15:02.692943 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:02.692892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:02.693150 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:02.692990 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:15:02.693336 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:02.693317 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7l9b5"] Oct 29 22:15:02.693430 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:02.693417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:02.693518 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:02.693497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:15:03.977908 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:03.977834 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:03.977908 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:03.977859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:03.978382 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:03.977834 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:03.978382 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:03.977936 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:15:03.978382 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:03.978012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:15:03.978382 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:03.978117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:15:05.978291 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:05.978261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:05.978291 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:05.978281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:05.978905 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:05.978265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:05.978905 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:05.978372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmv7w" podUID="d9e01536-6930-49aa-8507-203280a663f6" Oct 29 22:15:05.978905 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:05.978425 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rk9gl" podUID="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" Oct 29 22:15:05.978905 ip-10-0-139-29 kubenswrapper[2577]: E1029 22:15:05.978499 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7l9b5" podUID="12e17b5f-8baa-4708-8bef-2660e45d479c" Oct 29 22:15:07.675821 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.675790 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeReady" Oct 29 22:15:07.676205 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.675903 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:15:07.763393 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.763359 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7xrmb"] Oct 29 22:15:07.765322 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.765301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:07.766160 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.766099 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j8cft"] Oct 29 22:15:07.767801 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.767780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:07.784503 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.784485 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:15:07.784598 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.784514 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:15:07.784598 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.784564 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:15:07.784678 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.784606 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:15:07.784678 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.784641 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h6g72\"" Oct 29 22:15:07.797424 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.797406 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qnnrl\"" Oct 29 22:15:07.797424 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.797411 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:15:07.798321 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.798303 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n69kw"] Oct 29 22:15:07.801149 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.801131 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7xrmb"] Oct 29 22:15:07.801243 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.801161 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j8cft"] Oct 29 22:15:07.801243 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.801225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:07.817436 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.817417 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n69kw"] Oct 29 22:15:07.885201 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.885180 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:15:07.885314 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.885258 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:15:07.885374 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.885348 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9x6m\"" Oct 29 22:15:07.885374 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.885355 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:15:07.885374 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.885349 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:15:07.922647 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b5a282-9fbb-45af-93b8-41c247e552f8-data-volume\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:07.922765 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6844877d-8910-4d77-a31c-ca119d4f36ab-cert\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:07.922765 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn97\" (UniqueName: \"kubernetes.io/projected/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-api-access-lcn97\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:07.922870 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71169ebf-0176-4dec-8ac5-8a5adf441324-tmp-dir\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:07.922870 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2mh\" (UniqueName: \"kubernetes.io/projected/6844877d-8910-4d77-a31c-ca119d4f36ab-kube-api-access-6k2mh\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:07.922870 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5a282-9fbb-45af-93b8-41c247e552f8-crio-socket\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:07.922968 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71169ebf-0176-4dec-8ac5-8a5adf441324-config-volume\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:07.922968 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71169ebf-0176-4dec-8ac5-8a5adf441324-metrics-tls\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:07.923047 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.922991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:07.923085 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.923047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m27b5\" (UniqueName: \"kubernetes.io/projected/71169ebf-0176-4dec-8ac5-8a5adf441324-kube-api-access-m27b5\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:07.923085 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.923064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9b5a282-9fbb-45af-93b8-41c247e552f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:07.977984 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.977931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:07.977984 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.977957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:07.978113 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.978099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:07.993368 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.993339 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:15:07.993467 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.993451 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:15:07.993882 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.993845 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:15:07.993970 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.993907 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:15:07.994155 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.994135 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:15:07.994692 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.994667 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:15:07.994955 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:07.994924 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:15:08.023324 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2mh\" (UniqueName: \"kubernetes.io/projected/6844877d-8910-4d77-a31c-ca119d4f36ab-kube-api-access-6k2mh\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:08.023395 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5a282-9fbb-45af-93b8-41c247e552f8-crio-socket\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.023395 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71169ebf-0176-4dec-8ac5-8a5adf441324-config-volume\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.023395 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71169ebf-0176-4dec-8ac5-8a5adf441324-metrics-tls\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.023525 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.023525 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m27b5\" (UniqueName: \"kubernetes.io/projected/71169ebf-0176-4dec-8ac5-8a5adf441324-kube-api-access-m27b5\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.023621 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9b5a282-9fbb-45af-93b8-41c247e552f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.023621 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b5a282-9fbb-45af-93b8-41c247e552f8-data-volume\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.023748 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5a282-9fbb-45af-93b8-41c247e552f8-crio-socket\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.023748 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6844877d-8910-4d77-a31c-ca119d4f36ab-cert\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:08.023748 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn97\" (UniqueName: \"kubernetes.io/projected/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-api-access-lcn97\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.023897 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71169ebf-0176-4dec-8ac5-8a5adf441324-tmp-dir\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.024173 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.023941 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b5a282-9fbb-45af-93b8-41c247e552f8-data-volume\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.024173 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.024013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71169ebf-0176-4dec-8ac5-8a5adf441324-tmp-dir\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.024173 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.024053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.024173 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.024074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71169ebf-0176-4dec-8ac5-8a5adf441324-config-volume\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.026995 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.026976 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9b5a282-9fbb-45af-93b8-41c247e552f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.026995 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.026986 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71169ebf-0176-4dec-8ac5-8a5adf441324-metrics-tls\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.027220 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.027200 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6844877d-8910-4d77-a31c-ca119d4f36ab-cert\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:08.043451 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.043428 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m27b5\" (UniqueName: \"kubernetes.io/projected/71169ebf-0176-4dec-8ac5-8a5adf441324-kube-api-access-m27b5\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.043582 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.043561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn97\" (UniqueName: \"kubernetes.io/projected/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-api-access-lcn97\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.043869 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.043854 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2mh\" (UniqueName: \"kubernetes.io/projected/6844877d-8910-4d77-a31c-ca119d4f36ab-kube-api-access-6k2mh\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:08.077668 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.077647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:08.082073 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.082058 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:08.112929 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.112894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:08.393895 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.393622 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j8cft"] Oct 29 22:15:08.398057 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:08.398030 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6844877d_8910_4d77_a31c_ca119d4f36ab.slice/crio-ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983 WatchSource:0}: Error finding container ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983: Status 404 returned error can't find the container with id ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983 Oct 29 22:15:08.406133 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.406111 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7xrmb"] Oct 29 22:15:08.409000 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:08.408977 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71169ebf_0176_4dec_8ac5_8a5adf441324.slice/crio-618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872 WatchSource:0}: Error finding container 618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872: Status 404 returned error can't find the container with id 618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872 Oct 29 22:15:08.443218 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:08.443061 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n69kw"] Oct 29 22:15:08.446167 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:08.446145 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b5a282_9fbb_45af_93b8_41c247e552f8.slice/crio-02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210 WatchSource:0}: Error finding container 02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210: Status 404 returned error can't find the container with id 02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210 Oct 29 22:15:09.319638 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:09.319599 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j8cft" event={"ID":"6844877d-8910-4d77-a31c-ca119d4f36ab","Type":"ContainerStarted","Data":"ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983"} Oct 29 22:15:09.321026 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:09.320994 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7xrmb" event={"ID":"71169ebf-0176-4dec-8ac5-8a5adf441324","Type":"ContainerStarted","Data":"618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872"} Oct 29 22:15:09.324692 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:09.324670 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n69kw" event={"ID":"a9b5a282-9fbb-45af-93b8-41c247e552f8","Type":"ContainerStarted","Data":"9d2689cc65183d8f07371a597d494f81ded4e93b1d403a19680e9da84cf1dc1b"} Oct 29 22:15:09.324820 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:09.324733 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n69kw" event={"ID":"a9b5a282-9fbb-45af-93b8-41c247e552f8","Type":"ContainerStarted","Data":"fdc90ebc0bc6cc9d7e7ed549c1faaff15fc70fce1eaca4384fbf72afe0be9cac"} Oct 29 22:15:09.324820 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:09.324759 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n69kw" event={"ID":"a9b5a282-9fbb-45af-93b8-41c247e552f8","Type":"ContainerStarted","Data":"02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210"} Oct 29 22:15:10.330870 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.330839 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7xrmb" event={"ID":"71169ebf-0176-4dec-8ac5-8a5adf441324","Type":"ContainerStarted","Data":"176939d2354bfcca6be2c79b1fe09d99b2c81ee3a0cf2e15afd486cb4f1ec7c7"} Oct 29 22:15:10.331179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.330903 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7xrmb" event={"ID":"71169ebf-0176-4dec-8ac5-8a5adf441324","Type":"ContainerStarted","Data":"4b0e82f1d58c6455d3e72bc222ee8826122c7c0c23ed2b364b4e53ef3f41f2e0"} Oct 29 22:15:10.331645 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.331626 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:10.380442 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.380339 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7xrmb" podStartSLOduration=1.694936513 podStartE2EDuration="3.380322045s" totalImagesPullingTime="1.685385532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:10.378410235 +0000 UTC m=+48.996647682" watchObservedRunningTime="2026-10-29 22:15:10.380322045 +0000 UTC m=+48.998559485" Oct 29 22:15:10.714030 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.714004 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t5ml4"] Oct 29 22:15:10.716758 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.716742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.748832 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.748810 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:15:10.749010 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.748987 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28glh\"" Oct 29 22:15:10.749173 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.749155 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:15:10.768306 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.768284 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:15:10.768418 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.768366 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:15:10.773426 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.773409 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:15:10.773714 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.773664 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:15:10.844131 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-textfile\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-sys\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844261 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-root\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844391 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg595\" (UniqueName: \"kubernetes.io/projected/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-kube-api-access-xg595\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844391 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-wtmp\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844451 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844451 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-tls\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.844534 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.844465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-metrics-client-ca\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945421 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-tls\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945421 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-metrics-client-ca\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945596 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945596 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-textfile\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945596 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-sys\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945767 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-root\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945767 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg595\" (UniqueName: \"kubernetes.io/projected/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-kube-api-access-xg595\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945767 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945648 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-sys\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945767 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-wtmp\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945767 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945717 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-root\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.945767 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.946046 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945842 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-wtmp\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.946046 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.945849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-textfile\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.946189 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.946169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-metrics-client-ca\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.946251 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.946186 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.948332 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.948309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.952089 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.952060 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-tls\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:10.965691 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:10.965671 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg595\" (UniqueName: \"kubernetes.io/projected/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-kube-api-access-xg595\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:11.046973 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:11.046939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:11.065053 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:11.065028 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcdc9c9_6f01_44e9_a874_bf914bdd5c9f.slice/crio-533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea WatchSource:0}: Error finding container 533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea: Status 404 returned error can't find the container with id 533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea Oct 29 22:15:11.335605 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:11.335571 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n69kw" event={"ID":"a9b5a282-9fbb-45af-93b8-41c247e552f8","Type":"ContainerStarted","Data":"e83f2286dd239fc7f38bb1aea15fe868999f3813a25db3420fdb248003ae3e44"} Oct 29 22:15:11.337096 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:11.337070 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j8cft" event={"ID":"6844877d-8910-4d77-a31c-ca119d4f36ab","Type":"ContainerStarted","Data":"6aa7a20461abbdb614494edc85e10da470d14e34de25370f53582a95d551e5a6"} Oct 29 22:15:11.338226 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:11.338205 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ml4" event={"ID":"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f","Type":"ContainerStarted","Data":"533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea"} Oct 29 22:15:11.376260 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:11.376191 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n69kw" podStartSLOduration=2.291924128 podStartE2EDuration="4.376180297s" totalImagesPullingTime="2.084256169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:15:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:11.376014411 +0000 UTC m=+49.994251847" watchObservedRunningTime="2026-10-29 22:15:11.376180297 +0000 UTC m=+49.994417733" Oct 29 22:15:11.415818 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:11.415741 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j8cft" podStartSLOduration=1.74535997 podStartE2EDuration="4.415732531s" totalImagesPullingTime="2.670372561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:11.414999443 +0000 UTC m=+50.033236881" watchObservedRunningTime="2026-10-29 22:15:11.415732531 +0000 UTC m=+50.033969970" Oct 29 22:15:12.133637 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.133569 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:12.135711 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.135672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.151861 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151832 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:15:12.151861 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151846 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151863 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151863 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151887 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151897 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151927 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.151964 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:15:12.152005 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.152004 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-thfrk\"" Oct 29 22:15:12.152197 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.152028 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:15:12.166163 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.166141 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:12.254286 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254394 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254394 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254394 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254504 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254504 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254504 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254584 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254584 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254643 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254643 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254721 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.254756 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.254746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.342447 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.342414 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" containerID="f315c38c227e158324c5e4723251f56831f0dc9b2d2511c98d168fa00f4f852b" exitCode=0 Oct 29 22:15:12.342811 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.342446 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ml4" event={"ID":"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f","Type":"ContainerDied","Data":"f315c38c227e158324c5e4723251f56831f0dc9b2d2511c98d168fa00f4f852b"} Oct 29 22:15:12.355604 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355680 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355737 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355769 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355809 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355842 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355884 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.355884 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.355877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356033 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356084 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356136 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356136 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356226 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.356996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.356943 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360058 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.359494 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360058 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.359880 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360058 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.360032 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360573 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.360532 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360715 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.360675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360801 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.360786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.360950 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.360927 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.361115 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.361096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.361384 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.361368 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.375098 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.375067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.452413 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.452394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:12.768031 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:12.768003 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:12.770234 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:12.770210 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3386f2_acfa_4a49_b1a2_1589a13adf52.slice/crio-450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451 WatchSource:0}: Error finding container 450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451: Status 404 returned error can't find the container with id 450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451 Oct 29 22:15:13.347338 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:13.347297 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451"} Oct 29 22:15:13.349694 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:13.349656 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ml4" event={"ID":"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f","Type":"ContainerStarted","Data":"e1fcb857727df676db8ef3667c7f86d04975ff674cc76be027dea87441a25bf6"} Oct 29 22:15:13.349837 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:13.349728 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ml4" event={"ID":"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f","Type":"ContainerStarted","Data":"780202c83d4d8824d660bdc8f25bcd4a39b46906a264bce6e6f64a74cb00ac07"} Oct 29 22:15:14.354621 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:14.354589 2577 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d" exitCode=0 Oct 29 22:15:14.355077 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:14.354683 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d"} Oct 29 22:15:14.430676 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:14.430609 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t5ml4" podStartSLOduration=3.572323984 podStartE2EDuration="4.430590412s" totalImagesPullingTime="791.078948ms" totalInitContainerRuntime="67.18748ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:13.391068244 +0000 UTC m=+52.009305681" watchObservedRunningTime="2026-10-29 22:15:14.430590412 +0000 UTC m=+53.048827850" Oct 29 22:15:15.013001 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.012966 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf4d96c-bcdwj"] Oct 29 22:15:15.014905 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.014888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.031099 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.031073 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 22:15:15.031255 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.031139 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 22:15:15.031255 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.031150 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 22:15:15.044160 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.043879 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rfsln\"" Oct 29 22:15:15.044160 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.044014 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 22:15:15.044160 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.043882 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 22:15:15.044160 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.044031 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 22:15:15.044160 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.044135 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 22:15:15.071436 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.071401 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 29 22:15:15.072251 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.072224 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf4d96c-bcdwj"] Oct 29 22:15:15.180811 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.180773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.180996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.180828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.180996 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.180890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.181100 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.181012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.181100 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.181053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.181100 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.181086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.181238 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.181142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.181238 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.181166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.230274 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.230241 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-684967c95f-cf5zb"] Oct 29 22:15:15.232127 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.232112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.247938 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.247915 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 29 22:15:15.248081 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.247951 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 29 22:15:15.248081 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.247952 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:15:15.248081 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.248071 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 29 22:15:15.248168 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.248136 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-c2grw\"" Oct 29 22:15:15.248211 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.248200 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-657mjem3c3csp\"" Oct 29 22:15:15.253806 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.253781 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-684967c95f-cf5zb"] Oct 29 22:15:15.281950 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.281883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.281950 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.281915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.282424 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.282558 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.282558 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.282558 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.282722 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.282722 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.283256 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.282858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.283256 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.283217 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.283682 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.283455 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.283682 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.283471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.283682 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.283574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.284796 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.284672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.285434 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.285413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.306677 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.306649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.327382 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.327337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:15.383422 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bkf\" (UniqueName: \"kubernetes.io/projected/29897fb3-27ec-4f39-8f74-f96c5cba04a1-kube-api-access-g2bkf\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.384535 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/29897fb3-27ec-4f39-8f74-f96c5cba04a1-audit-log\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.384535 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-metrics-server-audit-profiles\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.384535 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-client-certs\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.384535 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-tls\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.384535 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.384535 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.383657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-client-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484310 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bkf\" (UniqueName: \"kubernetes.io/projected/29897fb3-27ec-4f39-8f74-f96c5cba04a1-kube-api-access-g2bkf\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484470 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/29897fb3-27ec-4f39-8f74-f96c5cba04a1-audit-log\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484470 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-metrics-server-audit-profiles\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484470 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-client-certs\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484470 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-tls\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484470 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.484470 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.484458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-client-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.485690 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.485091 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/29897fb3-27ec-4f39-8f74-f96c5cba04a1-audit-log\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.485690 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.485563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-metrics-server-audit-profiles\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.485690 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.485627 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.487587 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.487559 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-client-certs\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.487833 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.487769 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-tls\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.487833 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.487774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-client-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.507161 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.507132 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bkf\" (UniqueName: \"kubernetes.io/projected/29897fb3-27ec-4f39-8f74-f96c5cba04a1-kube-api-access-g2bkf\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.544050 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.543970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:15.628628 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.628597 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf4d96c-bcdwj"] Oct 29 22:15:15.633310 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:15.633284 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0c6f8b_6b6f_477d_b399_59c9bb3cb1ae.slice/crio-873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e WatchSource:0}: Error finding container 873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e: Status 404 returned error can't find the container with id 873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e Oct 29 22:15:15.839929 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:15.839900 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-684967c95f-cf5zb"] Oct 29 22:15:15.848355 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:15.848329 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29897fb3_27ec_4f39_8f74_f96c5cba04a1.slice/crio-21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e WatchSource:0}: Error finding container 21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e: Status 404 returned error can't find the container with id 21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e Oct 29 22:15:16.378969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.378638 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3"} Oct 29 22:15:16.378969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.378787 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f"} Oct 29 22:15:16.378969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.378830 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef"} Oct 29 22:15:16.378969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.378869 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618"} Oct 29 22:15:16.378969 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.378908 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71"} Oct 29 22:15:16.380720 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.380634 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" event={"ID":"29897fb3-27ec-4f39-8f74-f96c5cba04a1","Type":"ContainerStarted","Data":"21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e"} Oct 29 22:15:16.383865 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:16.383832 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf4d96c-bcdwj" event={"ID":"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae","Type":"ContainerStarted","Data":"873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e"} Oct 29 22:15:17.390984 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:17.390949 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerStarted","Data":"02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a"} Oct 29 22:15:17.478415 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:17.478363 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.509477662 podStartE2EDuration="5.478346911s" totalImagesPullingTime="2.955738321s" totalInitContainerRuntime="13.130928ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:17.478193681 +0000 UTC m=+56.096431117" watchObservedRunningTime="2026-10-29 22:15:17.478346911 +0000 UTC m=+56.096584342" Oct 29 22:15:18.394885 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:18.394849 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" event={"ID":"29897fb3-27ec-4f39-8f74-f96c5cba04a1","Type":"ContainerStarted","Data":"ef8e675049f92a1a626e48e0e8563e7a3851f3291dfdd0ce430b95a1d22727e4"} Oct 29 22:15:18.438691 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:18.438227 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" podStartSLOduration=1.964379322 podStartE2EDuration="3.438207385s" totalImagesPullingTime="1.473828063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:18.436145516 +0000 UTC m=+57.054382954" watchObservedRunningTime="2026-10-29 22:15:18.438207385 +0000 UTC m=+57.056444823" Oct 29 22:15:19.399946 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:19.399914 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf4d96c-bcdwj" event={"ID":"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae","Type":"ContainerStarted","Data":"47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a"} Oct 29 22:15:19.443610 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:19.443559 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bf4d96c-bcdwj" podStartSLOduration=2.6784956749999997 podStartE2EDuration="5.443544447s" totalImagesPullingTime="2.765048772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:19.443334691 +0000 UTC m=+58.061572125" watchObservedRunningTime="2026-10-29 22:15:19.443544447 +0000 UTC m=+58.061781878" Oct 29 22:15:20.374211 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:20.374182 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:21.344971 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:21.344940 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:23.843338 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.843303 2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf4d96c-bcdwj"] Oct 29 22:15:23.871002 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.870965 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67b8fc9b66-29q5f"] Oct 29 22:15:23.874066 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.874049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.895892 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.895855 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b8fc9b66-29q5f"] Oct 29 22:15:23.953334 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953503 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953503 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953503 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953648 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953648 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953648 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:23.953790 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:23.953659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.054916 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.054886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.054916 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.054922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.054942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.054976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.054992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055179 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055547 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055787 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055765 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055854 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055764 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055936 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055912 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.055970 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.055923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.058071 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.058052 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.058144 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.058054 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.075614 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.075583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.183693 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.183618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:24.476164 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:24.476087 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b8fc9b66-29q5f"] Oct 29 22:15:24.478956 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:24.478931 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ac81cd_cb0f_49aa_b0d9_067dff8442b0.slice/crio-f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d WatchSource:0}: Error finding container f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d: Status 404 returned error can't find the container with id f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d Oct 29 22:15:25.328562 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:25.328523 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:25.421560 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:25.421527 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b8fc9b66-29q5f" event={"ID":"34ac81cd-cb0f-49aa-b0d9-067dff8442b0","Type":"ContainerStarted","Data":"cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073"} Oct 29 22:15:25.421778 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:25.421575 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b8fc9b66-29q5f" event={"ID":"34ac81cd-cb0f-49aa-b0d9-067dff8442b0","Type":"ContainerStarted","Data":"f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d"} Oct 29 22:15:25.463296 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:25.463238 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67b8fc9b66-29q5f" podStartSLOduration=2.463223417 podStartE2EDuration="2.463223417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:25.462551444 +0000 UTC m=+64.080788881" watchObservedRunningTime="2026-10-29 22:15:25.463223417 +0000 UTC m=+64.081460886" Oct 29 22:15:27.684800 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.684755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:27.685251 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.684821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:27.713241 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.713202 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:15:27.713241 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.713230 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:15:27.717508 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.717472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:27.717761 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.717517 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:27.786204 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.786161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:27.800956 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.800924 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:15:27.803340 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.803324 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:15:27.809623 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.809603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:27.815242 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.815225 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:15:27.821578 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.821557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:27.821861 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.821838 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:15:27.831153 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:27.831128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:28.105838 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.105808 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vmv7w"] Oct 29 22:15:28.110068 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.110046 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:15:28.113309 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:28.113283 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e01536_6930_49aa_8507_203280a663f6.slice/crio-18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571 WatchSource:0}: Error finding container 18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571: Status 404 returned error can't find the container with id 18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571 Oct 29 22:15:28.115202 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.115181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:28.118961 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.118935 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rk9gl"] Oct 29 22:15:28.125175 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:28.125119 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2768c741_ba6f_4f8f_b20a_ac431cd8b83f.slice/crio-d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3 WatchSource:0}: Error finding container d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3: Status 404 returned error can't find the container with id d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3 Oct 29 22:15:28.407826 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.407741 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7l9b5"] Oct 29 22:15:28.410312 ip-10-0-139-29 kubenswrapper[2577]: W1029 22:15:28.410281 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e17b5f_8baa_4708_8bef_2660e45d479c.slice/crio-29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7 WatchSource:0}: Error finding container 29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7: Status 404 returned error can't find the container with id 29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7 Oct 29 22:15:28.432353 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.432329 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7l9b5" event={"ID":"12e17b5f-8baa-4708-8bef-2660e45d479c","Type":"ContainerStarted","Data":"29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7"} Oct 29 22:15:28.433421 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.433403 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rk9gl" event={"ID":"2768c741-ba6f-4f8f-b20a-ac431cd8b83f","Type":"ContainerStarted","Data":"d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3"} Oct 29 22:15:28.434436 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:28.434415 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmv7w" event={"ID":"d9e01536-6930-49aa-8507-203280a663f6","Type":"ContainerStarted","Data":"18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571"} Oct 29 22:15:29.442682 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:29.442640 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmv7w" event={"ID":"d9e01536-6930-49aa-8507-203280a663f6","Type":"ContainerStarted","Data":"9ae3d567d6d35bdf6144657a4ee76cf85e50700b14b576edcf991bdd5a66e0ff"} Oct 29 22:15:29.443092 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:29.442734 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmv7w" event={"ID":"d9e01536-6930-49aa-8507-203280a663f6","Type":"ContainerStarted","Data":"cb6c79a56667c06f387a87f946c8aa59824a54f8210a44e2b8fd1e4738c87606"} Oct 29 22:15:29.485809 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:29.485751 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vmv7w" podStartSLOduration=66.507077327 podStartE2EDuration="1m7.485734255s" totalImagesPullingTime="978.656928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:29.484319187 +0000 UTC m=+68.102556636" watchObservedRunningTime="2026-10-29 22:15:29.485734255 +0000 UTC m=+68.103971693" Oct 29 22:15:32.454269 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:32.454227 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7l9b5" event={"ID":"12e17b5f-8baa-4708-8bef-2660e45d479c","Type":"ContainerStarted","Data":"6997e4f0c03ac3260c23b04c7250b5735a2a9612e31ecd53289b3c9383ee9e59"} Oct 29 22:15:32.454692 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:32.454368 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:32.492639 ip-10-0-139-29 kubenswrapper[2577]: I1029 22:15:32.492575 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7l9b5" podStartSLOduration=67.4374591 podStartE2EDuration="1m10.49255902s" totalImagesPullingTime="3.05509992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:32.492535868 +0000 UTC m=+71.110773306" watchObservedRunningTime="2026-10-29 22:15:32.49255902 +0000 UTC m=+71.110796458" Oct 29 22:15:33.041055 ip-10-0-139-29 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:15:33.061106 ip-10-0-139-29 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:15:33.061274 ip-10-0-139-29 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:15:33.061448 ip-10-0-139-29 systemd[1]: kubelet.service: Consumed 3.801s CPU time. Oct 29 22:15:33.073344 ip-10-0-139-29 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:15:33.135417 ip-10-0-139-29 kubenswrapper[5367]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:15:33.135417 ip-10-0-139-29 kubenswrapper[5367]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:15:33.135417 ip-10-0-139-29 kubenswrapper[5367]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:15:33.135417 ip-10-0-139-29 kubenswrapper[5367]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:15:33.137540 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137458 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.137540 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137539 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.137540 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137543 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137547 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137551 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137555 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137559 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137562 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137565 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137573 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137576 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137579 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137582 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137584 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137587 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137589 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137592 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137595 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137597 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137600 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137603 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137605 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.137638 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137608 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137611 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137613 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137616 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137619 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137622 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137625 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137628 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137630 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137633 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137636 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137639 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137643 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137648 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137652 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137655 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137658 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137661 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137664 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.138185 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137667 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137670 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137673 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137675 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137678 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137680 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137683 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137686 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137689 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137691 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137709 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137715 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137720 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137723 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137726 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137728 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137731 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137734 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137737 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137739 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.138666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137742 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137745 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137747 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137752 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137754 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137757 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137759 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137761 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137764 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137767 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137769 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137773 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137776 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137779 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137782 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137784 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137787 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137789 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137792 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.139161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137795 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137797 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137800 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137803 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137805 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137808 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137811 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137814 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137817 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137821 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137824 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137827 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137829 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137832 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137834 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137837 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137839 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137842 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137844 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137847 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.139612 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137850 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137852 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137855 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137857 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137861 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.137870 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138646 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138653 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138657 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138660 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138663 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138667 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138671 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138675 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138679 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138682 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138686 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138689 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138692 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.140115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138710 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138714 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138717 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138720 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138723 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138725 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138728 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138731 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138734 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138736 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138739 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138743 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138746 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138750 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138752 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138755 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138758 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138760 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138763 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138767 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.140587 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138770 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138772 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138775 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138777 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138780 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138783 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138785 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138788 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138791 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138794 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138797 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138800 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138803 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138806 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138808 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138811 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138814 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138816 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138819 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138822 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.141094 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138824 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138827 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138829 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138832 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138834 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138837 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138840 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138842 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138845 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138848 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138850 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138853 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138856 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138859 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138861 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138864 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138867 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138869 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138872 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138875 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.141574 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138878 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138880 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138883 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138885 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138888 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138890 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138893 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138895 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138898 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138901 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138903 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138906 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138909 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138911 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138914 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138917 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138919 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138922 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138925 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.142107 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138927 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138930 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138933 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138936 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138938 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138940 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138944 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138946 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138949 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138952 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138954 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138957 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138959 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.138962 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139485 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139491 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139495 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139498 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139501 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.142611 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139503 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139506 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139509 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139512 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139515 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139517 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139520 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139523 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139525 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139533 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139536 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139539 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139541 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139544 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139547 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139550 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139552 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139555 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139557 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139561 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.143179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139564 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139566 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139570 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139572 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139575 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139578 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139581 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139583 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139586 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139590 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139592 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139595 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139597 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139600 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139602 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139605 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139607 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139610 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139613 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139615 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.143709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139618 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139622 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139625 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139628 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139631 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139635 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139639 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139642 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139645 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139648 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139651 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139653 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139662 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139664 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139667 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139670 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139672 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139675 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139677 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.144250 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139680 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139683 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139685 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139688 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139690 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139693 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139710 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139714 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139718 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139721 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139729 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139732 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139735 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139737 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139741 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139744 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139746 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139749 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139752 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.144730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139754 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139757 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139759 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139762 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139764 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139767 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139769 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139778 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139781 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139784 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139786 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139790 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139793 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139795 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139798 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139800 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139803 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139805 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139808 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139810 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.145227 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139813 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139815 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.139818 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139929 5367 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139947 5367 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139953 5367 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139958 5367 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139963 5367 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139967 5367 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139971 5367 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139976 5367 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139979 5367 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139982 5367 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139985 5367 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139989 5367 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139992 5367 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139995 5367 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.139998 5367 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140002 5367 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140005 5367 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140014 5367 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140023 5367 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140026 5367 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140029 5367 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:15:33.145731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140032 5367 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140036 5367 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140040 5367 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140043 5367 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140046 5367 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140050 5367 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140053 5367 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140056 5367 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140059 5367 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140062 5367 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140065 5367 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140071 5367 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140074 5367 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140077 5367 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140080 5367 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140084 5367 flags.go:64] FLAG: --enable-server="true" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140087 5367 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140094 5367 flags.go:64] FLAG: --event-burst="100" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140098 5367 flags.go:64] FLAG: --event-qps="50" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140101 5367 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140104 5367 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140107 5367 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140111 5367 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140114 5367 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140117 5367 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:15:33.146456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140121 5367 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140124 5367 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140127 5367 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140130 5367 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140138 5367 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140142 5367 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140144 5367 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140148 5367 flags.go:64] FLAG: --feature-gates="" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140152 5367 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140155 5367 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140158 5367 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140161 5367 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140164 5367 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140167 5367 flags.go:64] FLAG: --help="false" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140170 5367 flags.go:64] FLAG: --hostname-override="ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140173 5367 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140176 5367 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140179 5367 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140190 5367 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140194 5367 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140197 5367 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140200 5367 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140204 5367 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:15:33.147384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140207 5367 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140210 5367 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140213 5367 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140216 5367 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140219 5367 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140222 5367 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140225 5367 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140227 5367 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140230 5367 flags.go:64] FLAG: --lock-file="" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140233 5367 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140236 5367 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140239 5367 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140244 5367 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140247 5367 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140255 5367 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140259 5367 flags.go:64] FLAG: --logging-format="text" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140262 5367 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140265 5367 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140268 5367 flags.go:64] FLAG: --manifest-url="" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140271 5367 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140275 5367 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140278 5367 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140282 5367 flags.go:64] FLAG: --max-pods="110" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140285 5367 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140288 5367 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:15:33.147969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140291 5367 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140294 5367 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140298 5367 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140301 5367 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140304 5367 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140311 5367 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140314 5367 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140317 5367 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140320 5367 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140323 5367 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140326 5367 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140329 5367 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140332 5367 flags.go:64] FLAG: --port="10250" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140335 5367 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140338 5367 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01214bcd31e72ff2b" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140341 5367 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140344 5367 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140347 5367 flags.go:64] FLAG: --register-node="true" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140350 5367 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140354 5367 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140356 5367 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140359 5367 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140368 5367 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140372 5367 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140375 5367 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140378 5367 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:15:33.148568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140381 5367 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140384 5367 flags.go:64] FLAG: --runonce="false" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140387 5367 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140390 5367 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140393 5367 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140396 5367 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140399 5367 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140402 5367 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140406 5367 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140409 5367 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140412 5367 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140415 5367 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140418 5367 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140421 5367 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140424 5367 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140427 5367 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140429 5367 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140434 5367 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140437 5367 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140440 5367 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140447 5367 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140450 5367 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140453 5367 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140455 5367 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140458 5367 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:15:33.149212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140462 5367 flags.go:64] FLAG: --v="2" Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140466 5367 flags.go:64] FLAG: --version="false" Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140471 5367 flags.go:64] FLAG: --vmodule="" Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140474 5367 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.140483 5367 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140680 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140685 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140688 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140691 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140708 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140712 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140716 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140718 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140721 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140724 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140729 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140732 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140734 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140737 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140740 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140742 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.149866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140745 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140749 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140754 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140757 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140760 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140762 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140765 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140768 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140770 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140773 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140776 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140778 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140781 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140784 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140786 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140789 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140798 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140801 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140804 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.150380 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140807 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140810 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140812 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140815 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140818 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140820 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140823 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140827 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140830 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140832 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140835 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140837 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140840 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140844 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140848 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140853 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140856 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140859 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140862 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.151011 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140864 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140867 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140870 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140873 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140875 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140878 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140881 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140883 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140886 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140888 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140891 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140899 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140902 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140905 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140907 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140910 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140913 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140915 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140918 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.151576 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140921 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140925 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140931 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140934 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140936 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140939 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140942 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140944 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140947 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140951 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140953 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140956 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140959 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140962 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140964 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140967 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140970 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140972 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140975 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.152115 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140978 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140980 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140983 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140985 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140988 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140991 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.140999 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141002 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141005 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141007 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141010 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141012 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141015 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141017 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.141045 5367 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141097 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.152656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141102 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141105 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141108 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141110 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141113 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141116 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141118 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141123 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141126 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141128 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141131 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141134 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141136 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141139 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141142 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141144 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141147 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141150 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141152 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.153102 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141155 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141158 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141160 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141163 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141171 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141174 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141177 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141179 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141182 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141185 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141187 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141191 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141194 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141196 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141199 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141202 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141204 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141207 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141209 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141212 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.153605 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141217 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141219 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141222 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141225 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141227 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141230 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141234 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141238 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141241 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141244 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141246 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141249 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141251 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141254 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141257 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141266 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141269 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141276 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141279 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141282 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.154162 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141284 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141287 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141290 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141294 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141296 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141299 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141301 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141304 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141307 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141309 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141312 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141314 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141318 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141321 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141323 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141326 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141329 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141341 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141345 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.154931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141348 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141352 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141354 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141357 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141360 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141363 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141365 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141368 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141370 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141373 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141376 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141384 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141387 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141389 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141392 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141394 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141398 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141401 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141403 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141406 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.155417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141409 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141412 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141414 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141418 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141421 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141425 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.141428 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.145462 5367 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.145477 5367 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.145495 5367 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.145499 5367 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145560 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145566 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145569 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145572 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145576 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145578 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145582 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145584 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.155916 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145587 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145590 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145592 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145595 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145599 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145602 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145604 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145607 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145611 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145613 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145616 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145619 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145621 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145624 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145627 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145630 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145632 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145636 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145641 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.156366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145644 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145646 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145649 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145652 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145655 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145657 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145660 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145662 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145666 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145668 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145671 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145674 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145676 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145679 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145682 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145684 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145687 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145690 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145693 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145716 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.156877 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145719 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145722 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145725 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145728 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145730 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145733 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145736 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145739 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145742 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145744 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145747 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145750 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145753 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145755 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145758 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145760 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145763 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145766 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145768 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145771 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.157357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145774 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145777 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145780 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145782 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145785 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145787 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145790 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145793 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145796 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145799 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145801 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145805 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145808 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145810 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145813 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145815 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145818 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145821 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145823 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145827 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.157863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145830 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145832 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145835 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145837 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145840 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145843 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145845 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145848 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145851 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145854 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145857 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145859 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145862 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145866 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145870 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145874 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145877 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145880 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.158337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.145883 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.145911 5367 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146248 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146578 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146585 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146589 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146596 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146600 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146606 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146611 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146615 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146620 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146624 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146629 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146640 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146645 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.158793 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146650 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146655 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146660 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146664 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146669 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146673 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146677 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146683 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146687 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146692 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146717 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146727 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146732 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146736 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146742 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146747 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146751 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146755 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146760 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146764 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.159178 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146769 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146773 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146777 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146788 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146793 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146799 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146806 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146811 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146816 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146821 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146825 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146830 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146835 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146839 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146843 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146848 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146856 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146861 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146865 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.159791 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146870 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146874 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146878 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146883 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146887 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146891 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146896 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146900 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146905 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146909 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146918 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146923 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146928 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146932 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146937 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146941 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146946 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146951 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146955 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146960 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.160257 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146964 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146969 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146978 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146983 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146987 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146992 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.146996 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147001 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147005 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147010 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147014 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147018 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147022 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147027 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147031 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147040 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147044 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147048 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147052 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147058 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.160744 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147064 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147069 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147073 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147078 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147082 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147086 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147091 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147100 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147105 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147110 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147115 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147120 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147124 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.147186 5367 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147433 5367 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147450 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:33.161215 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147454 5367 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147457 5367 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147460 5367 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147464 5367 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147467 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147470 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147473 5367 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147476 5367 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147479 5367 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147481 5367 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147484 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147487 5367 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147490 5367 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147493 5367 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147496 5367 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147498 5367 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147501 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147505 5367 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147508 5367 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:33.161615 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147511 5367 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147514 5367 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147516 5367 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147519 5367 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147522 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147525 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147527 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147530 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147535 5367 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147541 5367 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147544 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147547 5367 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147550 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147553 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147555 5367 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147558 5367 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147561 5367 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147564 5367 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147566 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:33.162086 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147569 5367 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147572 5367 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147574 5367 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147579 5367 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147581 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147584 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147587 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147590 5367 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147592 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147596 5367 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147598 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147608 5367 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147611 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147614 5367 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147617 5367 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147619 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147622 5367 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147624 5367 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147627 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:33.162854 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147630 5367 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147633 5367 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147635 5367 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147638 5367 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147640 5367 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147643 5367 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147646 5367 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147648 5367 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147651 5367 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147654 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147656 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147659 5367 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147661 5367 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147664 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147666 5367 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147669 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147671 5367 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147674 5367 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147676 5367 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147679 5367 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:33.163517 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147681 5367 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147684 5367 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147687 5367 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147691 5367 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147717 5367 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147721 5367 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147725 5367 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147727 5367 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147730 5367 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147733 5367 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147735 5367 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147738 5367 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147741 5367 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147743 5367 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147746 5367 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147748 5367 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147751 5367 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147754 5367 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147756 5367 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:33.164068 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147759 5367 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147762 5367 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147765 5367 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147769 5367 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147772 5367 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147774 5367 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147777 5367 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:33.147779 5367 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.147842 5367 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.148828 5367 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.148876 5367 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.149055 5367 server.go:1014] "Starting client certificate rotation" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.149192 5367 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.149242 5367 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:09:21 +0000 UTC" deadline="2028-04-17 16:14:40.712286829 +0000 UTC" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.149269 5367 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12857h59m7.563019122s" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.149501 5367 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:15:33.164514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.150651 5367 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:15:33.164966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.151941 5367 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:15:33.164966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.153724 5367 log.go:25] "Validated CRI v1 image API" Oct 29 22:15:33.164966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.154472 5367 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:15:33.164966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.156619 5367 fs.go:135] Filesystem UUIDs: map[0a7997db-3caf-411d-9ca1-2a363cf23032:/dev/nvme0n1p4 67bb8026-ec41-4e0c-85ac-886b2d5c78dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 29 22:15:33.165101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.156634 5367 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12e17b5f-8baa-4708-8bef-2660e45d479c/volumes/kubernetes.io~projected/kube-api-access-lt97c:{mountpoint:/var/lib/kubelet/pods/12e17b5f-8baa-4708-8bef-2660e45d479c/volumes/kubernetes.io~projected/kube-api-access-lt97c major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/192ace5d-ce40-41f2-b494-4dd3bfae0507/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/192ace5d-ce40-41f2-b494-4dd3bfae0507/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volumes/kubernetes.io~projected/kube-api-access-hhqvh:{mountpoint:/var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volumes/kubernetes.io~projected/kube-api-access-hhqvh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2768c741-ba6f-4f8f-b20a-ac431cd8b83f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2768c741-ba6f-4f8f-b20a-ac431cd8b83f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~projected/kube-api-access-g2bkf:{mountpoint:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~projected/kube-api-access-g2bkf major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3189ad37-09b3-47dc-aa9e-e384d7a0ff21/volumes/kubernetes.io~projected/kube-api-access-nmftn:{mountpoint:/var/lib/kubelet/pods/3189ad37-09b3-47dc-aa9e-e384d7a0ff21/volumes/kubernetes.io~projected/kube-api-access-nmftn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~projected/kube-api-access-g98c2:{mountpoint:/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~projected/kube-api-access-g98c2 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~projected/kube-api-access-lcrdz:{mountpoint:/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~projected/kube-api-access-lcrdz major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/425b6b6b-e38b-4ceb-b473-f1c397eba5d0/volumes/kubernetes.io~projected/kube-api-access-lmmzt:{mountpoint:/var/lib/kubelet/pods/425b6b6b-e38b-4ceb-b473-f1c397eba5d0/volumes/kubernetes.io~projected/kube-api-access-lmmzt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6844877d-8910-4d77-a31c-ca119d4f36ab/volumes/kubernetes.io~projected/kube-api-access-6k2mh:{mountpoint:/var/lib/kubelet/pods/6844877d-8910-4d77-a31c-ca119d4f36ab/volumes/kubernetes.io~projected/kube-api-access-6k2mh major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6844877d-8910-4d77-a31c-ca119d4f36ab/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6844877d-8910-4d77-a31c-ca119d4f36ab/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/690683ad-1ece-45ed-82ae-373e0a359b61/volumes/kubernetes.io~projected/kube-api-access-lfbw4:{mountpoint:/var/lib/kubelet/pods/690683ad-1ece-45ed-82ae-373e0a359b61/volumes/kubernetes.io~projected/kube-api-access-lfbw4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/697eac89-c298-46e4-ac8c-bc825959b7bd/volumes/kubernetes.io~projected/kube-api-access-p7dwh:{mountpoint:/var/lib/kubelet/pods/697eac89-c298-46e4-ac8c-bc825959b7bd/volumes/kubernetes.io~projected/kube-api-access-p7dwh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71169ebf-0176-4dec-8ac5-8a5adf441324/volumes/kubernetes.io~projected/kube-api-access-m27b5:{mountpoint:/var/lib/kubelet/pods/71169ebf-0176-4dec-8ac5-8a5adf441324/volumes/kubernetes.io~projected/kube-api-access-m27b5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71169ebf-0176-4dec-8ac5-8a5adf441324/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/71169ebf-0176-4dec-8ac5-8a5adf441324/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~projected/kube-api-access-xg595:{mountpoint:/var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~projected/kube-api-access-xg595 major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~projected/kube-api-access-jtl7r:{mountpoint:/var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~projected/kube-api-access-jtl7r major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ab6261a-9f14-479b-9618-ed6672c8658d/volumes/kubernetes.io~projected/kube-api-access-ttnct:{mountpoint:/var/lib/kubelet/pods/9ab6261a-9f14-479b-9618-ed6672c8658d/volumes/kubernetes.io~projected/kube-api-access-ttnct major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9b5a282-9fbb-45af-93b8-41c247e552f8/volumes/kubernetes.io~projected/kube-api-access-lcn97:{mountpoint:/var/lib/kubelet/pods/a9b5a282-9fbb-45af-93b8-41c247e552f8/volumes/kubernetes.io~projected/kube-api-access-lcn97 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9b5a282-9fbb-45af-93b8-41c247e552f8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a9b5a282-9fbb-45af-93b8-41c247e552f8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb595d53-05aa-4575-a5e2-658dfcfbe833/volumes/kubernetes.io~projected/kube-api-access-6r5rh:{mountpoint:/var/lib/kubelet/pods/cb595d53-05aa-4575-a5e2-658dfcfbe833/volumes/kubernetes.io~projected/kube-api-access-6r5rh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9e01536-6930-49aa-8507-203280a663f6/volumes/kubernetes.io~projected/kube-api-access-r56ll:{mountpoint:/var/lib/kubelet/pods/d9e01536-6930-49aa-8507-203280a663f6/volumes/kubernetes.io~projected/kube-api-access-r56ll major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9e01536-6930-49aa-8507-203280a663f6/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d9e01536-6930-49aa-8507-203280a663f6/volumes/kubernetes.io~secret/metrics-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/alertmanager/9 major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/config-reloader/6 major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~empty-dir/config-out major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~projected/kube-api-access-94z6s:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~projected/kube-api-access-94z6s major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~projected/tls-assets major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/config-volume major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/web-config major:0 minor:264 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/20e6dce9d9333a34274b50165f3567471b256a04cc83203af41d0c5237841442/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a3204d8e4b500e5c30c801c1f084f823c7c6c854d7401242929beb942b391558/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/01feee17d2e3b5529cbfe8bee2a7cc694fd967302b535f5fd01fe90078b29dcc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cf3eaeb10a9b2cffd9fc415d011c77e6729623418367fd46ef8955c9753538c5/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dcdab8740f1ce08776099f190fd4e596fa989f91397763807d0f81911ad7bc5e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/47c77324bf6efe8fc1a3a0b4658ca0c87888f3afbcb319bd0432f41c6423f753/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2d551a9437236fcd86df51d59657b2ead1700fcae5bb57b2f8d0d3f7cce5c6fc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ba22229e8f68587f5c61980be48c10aa5d6babc84dd77a0eaf336be4ae90ba24/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/056a99fdbc4324b95615b184a8a70c5aa6d0f289189635f7d3987652d7791bba/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b348ceaeb91c8aa68acd2e699bf6933eb41b34d15790a55a6eead1abd147fe06/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/6d4db727cb38220dec521ff868ed2557f216c177b7df230b05d1ede1b54a6afd/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/69952635a75b3c1d1071b4b8c2753d95a6053318d2598c14c01fa41026e9f5c9/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/edc8f4d0766ee1178f104281f19dd1629a83ba3e5a37b81a3d077116d5865fb2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/bfec8780c9700a709536a7cd0882a8c85e9f742136cccc49ecb421786a0b72b3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/630ea7918f86c7e38852f4b678b56adf303d9d99eb2cb4fe48f4f778924cd7d2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/31e020f39cfda5566736f0b1683dbeb859070e9c0cc86c2a6d80b2313cb8edfe/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e1ce0366fb4cb8843452addf14270864523997e4395c2cec8f5cb0718c464bd0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bc68434fa34f114b86c571f477cad727c45fe09929a32774247e97fa44896eba/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/fbde9ead2ceedf34f69aa3b1dcfc8698be25d5b8d81f66d32a0db5b77a6e73c1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7da52257dfa988ce346905ff873d2789e48b1d3fc36cd84765282890c28c7b72/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/186a8a1a5c793b4aa21d3828e2ce09649c8515138fd0a81bb0112734b8c52243/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5094ddb1e9b100f21e03e27629f971c33da080e52c86c5a273ead4005633eb77/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/6bc0bd91b69f0cd898b1fe3f30453b540e81f35a9b44f35e999b5f8dbfbf8fae/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/9fc52089e63ccd80b8e74f9d28efb49475b9a2039b8395fe6d29c029d7dd8a7e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/0781d3409238e91ada3a11c85b34de30f05e50ac4fff960691b66e6b4ca94415/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b0ae9708654310e17369521eba6eee96b9e277defbd402572a81e6603d19c91b/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/153e13e52e8f0f624c9c80fbcccd0be1a6d403d833b0bc5cd23de4fc9f5f4333/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/c8aae6f82b4e76acef11d737dc04d50cc1c94286780af3e69393510c79d013a6/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/390476fdca7c671c6e5c4b36b3a5c01d035d86ffa676d1416c95b8d334dc8c3b/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/7fc9ea8666a78172b00b9314066e16718f266dd4c97868f55cfd410c44590923/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/23aea6c4a27b8978d7b56063ccd398c3a0c8e62c6748d93d436c21befd12fa70/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/805ac1dfebc1ef7732eb30e7debc30805417feca84cff9d1123fdc0ef496f2f9/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/8edeec767632e644d25efb27e253cef10cfd0e222d706aabf93a6c9a01bf7a6d/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/b2564aa22686b8e94afd3d61eab3faf180232bbfc83b0830344d5d1af20b8bb0/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e3d2759ae9d0dbb5d4493240caf683a183ce2f1c340c5da651b224528cd3a8ee/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/213c3ad5035dad26419a750233ab0d668f30844c4165e2fa812f65e6c97b6a58/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/6af8e3be297293c7c00f967280a821869a13f990ddef7ddf220591ee9b846d18/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/56d5e619795f342a1fdc5d9588c10077b028897a828b261da2c8909a02f17b2b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/24f8b80e018560994126820161f45d2979e2ff6ab23ff56e1a6d6626ed9ab5e8/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/5dd82ea7bfd634d28207468f0ec71b4b8adaaed5b065472b98e367afb8a0f077/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/12a303352cee8148daa1056023f8aae3df1bd1479b67171f1096141c5b54b4fb/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/3038f15592f77909e6ecc412fbbd1541de5fb21516d2e8e1f9ae923dfd2a4f69/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/28617f5a163251521889706b0523d37d10e67c4738c6fbdccd1a34b697154f1b/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/fed16660c15c1de04510bc09ce71ec0982035f2a48717007f19081ec87d2a228/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/08b0efbf0211a075fb81adef46a696072afbd54b52779534b2d7c122c7019a10/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/81bb08258606185b0e4f88cb0541a6e76eeb61bca8067e73f1b691f9d8490531/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/3c1eb699e3f62e3f8dc77d351d19966e101e407ca0eea031c556640ec52e9ef1/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/485d7376e11883758b1ddb97ff7bb9d920bb14309ff996f4c667045a5103a03b/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/3c4ca43858ff6ba2227e81b68b2fe3aa043fb3eb74e37c8e035cf7795316a9ae/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/f1d955230d0995fe36b182afb922363677ef71807028d9090a512dc830f26136/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/306c35b5a9026b426bdf14b3c39e8891cf753b7ae4155af4eeeb3177a1413b97/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/d6dd0646fb5fe81b21da056d67273dca614739b0a794ae82b7772b47d6a81653/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/a4916f944f3c450d3e37e3e85b01515425b4ca28bdf311ab67bfba6996f3398e/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5d650eb06a955718d5385527cfdbb7f08f7cf85a39e6515ce80f2d0bb7172a49/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b9ddddd478f70d55838d78dbe1aeeb206ada6693ab75c91c2a3b601582b129ce/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/96fca1beac60f42a00977f3b5f8b01002d565a25c78cfc95d4b2240a10f9f9a9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9bf8b2dc41a1593ef0a36b1eeffdda9340d787ee2386b3425551c17c277dbba6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/50d980fb238d1e935377dfe548f31810f47728bb657569bfff8a0c0581d0cead/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/bc7a7337768f616cf7f13e7372e4537626d60dbfaba06b48b56a87d04aa73dac/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/f9df794bc400f704f477dab6ccdad71e44b641c61e284e12a782ba75308e87ad/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0373af52be1d1194b957e122864ff158fe154b52001415f24ac0ffdd295fe3b3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/b140898aae1cf6dcf7b275f50ccba3099ebca423b14986eb96051fcba1cba146/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/334a5fbf1ec20cf61d52e420f038829b477b963ce89067aba20a23334bc183fa/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.164390 5367 manager.go:217] Machine: {Timestamp:2026-10-29 22:15:33.163934519 +0000 UTC m=+0.058009433 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083536 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2520a172c216540354154931caff7b SystemUUID:ec2520a1-72c2-1654-0354-154931caff7b BootID:0a412d36-53d8-4b42-b31d-6071e1eb9fea Filesystems:[{Device:/var/lib/kubelet/pods/192ace5d-ce40-41f2-b494-4dd3bfae0507/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/618c60323b3fc634811e44aa10cc694af9bbad254b5faed2196d7cd54906c872/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de02d03354570a85d69ab8e60917eef49f6f47c516c77d507815ec60adfbb66e/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volumes/kubernetes.io~projected/kube-api-access-hhqvh DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d011760d37303ebf191e5ba9852a81b6140ed51944024c048c0bf38ea622fff3/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:281 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71169ebf-0176-4dec-8ac5-8a5adf441324/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/425b6b6b-e38b-4ceb-b473-f1c397eba5d0/volumes/kubernetes.io~projected/kube-api-access-lmmzt DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~projected/kube-api-access-94z6s DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21a7b4a580ee6879c9215f4c8627c24bb5ec4213f16934b7164d9dad3d403a9e/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9e01536-6930-49aa-8507-203280a663f6/volumes/kubernetes.io~projected/kube-api-access-r56ll DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/533a1993f6c5dae78065aea0b3e2f47bc2f6f95deec730dd3c59bd2890a8a4ea/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~projected/kube-api-access-xg595 DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ab6261a-9f14-479b-9618-ed6672c8658d/volumes/kubernetes.io~projected/kube-api-access-ttnct DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:262 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~projected/kube-api-access-g2bkf DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~projected/kube-api-access-jtl7r DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ead5088147d1362606cf72b823bcb6c7db677cccc6b5fb9b2fee21e05368c983/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes/kubernetes.io~projected/kube-api-access-lcrdz DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/697eac89-c298-46e4-ac8c-bc825959b7bd/volumes/kubernetes.io~projected/kube-api-access-p7dwh DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:261 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12e17b5f-8baa-4708-8bef-2660e45d479c/volumes/kubernetes.io~projected/kube-api-access-lt97c DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ae00634b0c3c5afb41ca13d46f2b9827a0e1fd705838e3d89808030c753cd6d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2768c741-ba6f-4f8f-b20a-ac431cd8b83f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29f889c2d010ac06f1b2116e852d49097f63c4fc8a0673b96747c405cc3b1ff7/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6844877d-8910-4d77-a31c-ca119d4f36ab/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71169ebf-0176-4dec-8ac5-8a5adf441324/volumes/kubernetes.io~projected/kube-api-access-m27b5 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3189ad37-09b3-47dc-aa9e-e384d7a0ff21/volumes/kubernetes.io~projected/kube-api-access-nmftn DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05c1619c796ba3543bac38af2a3256bb284e21ea452e5a7652cedb9c6ee7136d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:263 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9b5a282-9fbb-45af-93b8-41c247e552f8/volumes/kubernetes.io~projected/kube-api-access-lcn97 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9e01536-6930-49aa-8507-203280a663f6/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18335428090fd3cd1fb8e66c04747a3bd5cf2efcad1eafeb7aa1c742c292b571/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb595d53-05aa-4575-a5e2-658dfcfbe833/volumes/kubernetes.io~projected/kube-api-access-6r5rh DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02353c4e51bfc14753979b66fc5e487781fd0cdc6a519136cb7a17fcd3373210/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c97874c7298dbd78138d7678673702f74e41a634bad7e2306cf64a255e15c4ed/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ef3d91b7312e81f2e7a41a0ce617b3c1e10965debf420eb2ca068307daac239/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9b5a282-9fbb-45af-93b8-41c247e552f8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29897fb3-27ec-4f39-8f74-f96c5cba04a1/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes/kubernetes.io~projected/kube-api-access-g98c2 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/690683ad-1ece-45ed-82ae-373e0a359b61/volumes/kubernetes.io~projected/kube-api-access-lfbw4 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93949f750a8c5d145a2992f4995865214f5a29531a124b2beedd3e935ced307d/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83fffeb4-7835-4126-8258-a12f509a443f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6844877d-8910-4d77-a31c-ca119d4f36ab/volumes/kubernetes.io~projected/kube-api-access-6k2mh DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d630f4beb614edb625b39c2225d3d61a4d7a0380e7502ff6ee595e98c1c23636/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22c03e26-3445-48ed-9296-8678cb720276/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abb1c249fa8d4d71b344498419b39247dc93dec4342bd4118e6b796df2a9bdd8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02353c4e51bfc14 MacAddress:b2:b3:d4:4e:f0:5d Speed:10000 Mtu:8901} {Name:18335428090fd3c MacAddress:16:f3:f7:54:ae:95 Speed:10000 Mtu:8901} {Name:21a7b4a580ee687 MacAddress:72:ba:f9:41:f8:bb Speed:10000 Mtu:8901} {Name:29f889c2d010ac0 MacAddress:6e:9a:5e:01:17:da Speed:10000 Mtu:8901} {Name:450512304ed638d MacAddress:c6:aa:eb:4d:38:22 Speed:10000 Mtu:8901} {Name:618c60323b3fc63 MacAddress:d2:7e:52:25:2a:d9 Speed:10000 Mtu:8901} {Name:873256a1837b3b3 MacAddress:3e:22:88:94:be:69 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:75:5f:7f Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:33:76:0b:55:cb Speed:0 Mtu:8901} {Name:d011760d37303eb MacAddress:3e:62:cf:d8:bd:94 Speed:10000 Mtu:8901} {Name:ead5088147d1362 MacAddress:12:95:c1:4c:48:1d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:75:5f:7f Speed:0 Mtu:9001} {Name:f30987dc08cddbe MacAddress:86:00:f0:7f:d2:25 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b6:3c:3e:0a:8b:9e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:77:f9:5e:36:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.164764 5367 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.164811 5367 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165101 5367 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165128 5367 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165255 5367 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165262 5367 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165272 5367 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:15:33.165340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165285 5367 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:15:33.165728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165384 5367 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:15:33.165728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165549 5367 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:15:33.165728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165660 5367 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:15:33.165728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165713 5367 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:15:33.165838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165728 5367 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:15:33.165838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165749 5367 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:15:33.165838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165764 5367 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:15:33.165838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.165779 5367 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:15:33.166624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.166608 5367 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:15:33.166661 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.166625 5367 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:15:33.167130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.167106 5367 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:15:33.169439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169425 5367 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:15:33.169439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169439 5367 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:15:33.169566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169524 5367 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:15:33.169789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169732 5367 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:15:33.169789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169763 5367 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:15:33.169876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169854 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:15:33.169876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169865 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:15:33.169876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169871 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:15:33.169876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169877 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169882 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169888 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169894 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169899 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169906 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169913 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169921 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169930 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169956 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:15:33.169997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.169961 5367 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:15:33.173241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.173228 5367 server.go:1263] "Started kubelet" Oct 29 22:15:33.178323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.173296 5367 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:15:33.178323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.173398 5367 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:15:33.178323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.173828 5367 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:15:33.178323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.175853 5367 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:15:33.173762 ip-10-0-139-29 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:15:33.178684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.178437 5367 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:15:33.178684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.178476 5367 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:15:33.180026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.178837 5367 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:23 +0000 UTC" deadline="2028-08-01 02:37:30.72120405 +0000 UTC" Oct 29 22:15:33.180026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.178887 5367 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15388h21m57.54232061s" Oct 29 22:15:33.180026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.179017 5367 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:15:33.180026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.179020 5367 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:15:33.180026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.179039 5367 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:15:33.180026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.179027 5367 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:15:33.180339 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:33.180276 5367 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found" Oct 29 22:15:33.181219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.181200 5367 factory.go:55] Registering systemd factory Oct 29 22:15:33.181219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.181222 5367 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.182599 5367 factory.go:153] Registering CRI-O factory Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.182616 5367 factory.go:223] Registration of the crio container factory successfully Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.182670 5367 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.182687 5367 factory.go:103] Registering Raw factory Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.182721 5367 manager.go:1199] Started watching for new ooms in manager Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.183016 5367 manager.go:319] Starting recovery of all containers Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.183946 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.183994 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="690683ad-1ece-45ed-82ae-373e0a359b61" volumeName="kubernetes.io/projected/690683ad-1ece-45ed-82ae-373e0a359b61-kube-api-access-lfbw4" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184009 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb595d53-05aa-4575-a5e2-658dfcfbe833" volumeName="kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-binary-copy" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184024 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184037 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3189ad37-09b3-47dc-aa9e-e384d7a0ff21" volumeName="kubernetes.io/projected/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-kube-api-access-nmftn" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184049 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184087 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184111 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184129 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fffeb4-7835-4126-8258-a12f509a443f" volumeName="kubernetes.io/projected/83fffeb4-7835-4126-8258-a12f509a443f-kube-api-access-jtl7r" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184145 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e17b5f-8baa-4708-8bef-2660e45d479c" volumeName="kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184158 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22c03e26-3445-48ed-9296-8678cb720276" volumeName="kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184171 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184185 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184198 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5a282-9fbb-45af-93b8-41c247e552f8" volumeName="kubernetes.io/projected/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-api-access-lcn97" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184213 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9e01536-6930-49aa-8507-203280a663f6" volumeName="kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184257 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184270 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fffeb4-7835-4126-8258-a12f509a443f" volumeName="kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-tmp" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184284 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22c03e26-3445-48ed-9296-8678cb720276" volumeName="kubernetes.io/projected/22c03e26-3445-48ed-9296-8678cb720276-kube-api-access-hhqvh" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184297 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184310 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184323 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184346 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5a282-9fbb-45af-93b8-41c247e552f8" volumeName="kubernetes.io/secret/a9b5a282-9fbb-45af-93b8-41c247e552f8-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184361 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb595d53-05aa-4575-a5e2-658dfcfbe833" volumeName="kubernetes.io/projected/cb595d53-05aa-4575-a5e2-658dfcfbe833-kube-api-access-6r5rh" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184373 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184387 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71169ebf-0176-4dec-8ac5-8a5adf441324" volumeName="kubernetes.io/secret/71169ebf-0176-4dec-8ac5-8a5adf441324-metrics-tls" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184399 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/projected/29897fb3-27ec-4f39-8f74-f96c5cba04a1-kube-api-access-g2bkf" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184413 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184427 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6844877d-8910-4d77-a31c-ca119d4f36ab" volumeName="kubernetes.io/secret/6844877d-8910-4d77-a31c-ca119d4f36ab-cert" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184444 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184459 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184471 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184484 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184496 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425b6b6b-e38b-4ceb-b473-f1c397eba5d0" volumeName="kubernetes.io/projected/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-kube-api-access-lmmzt" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184509 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="697eac89-c298-46e4-ac8c-bc825959b7bd" volumeName="kubernetes.io/configmap/697eac89-c298-46e4-ac8c-bc825959b7bd-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184522 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184534 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71169ebf-0176-4dec-8ac5-8a5adf441324" volumeName="kubernetes.io/projected/71169ebf-0176-4dec-8ac5-8a5adf441324-kube-api-access-m27b5" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184549 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5a282-9fbb-45af-93b8-41c247e552f8" volumeName="kubernetes.io/empty-dir/a9b5a282-9fbb-45af-93b8-41c247e552f8-data-volume" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184563 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb595d53-05aa-4575-a5e2-658dfcfbe833" volumeName="kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184575 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184589 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22c03e26-3445-48ed-9296-8678cb720276" volumeName="kubernetes.io/secret/22c03e26-3445-48ed-9296-8678cb720276-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184602 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184615 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184628 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" volumeName="kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-tls" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184642 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" volumeName="kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184655 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184668 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184681 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22c03e26-3445-48ed-9296-8678cb720276" volumeName="kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-env-overrides" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184693 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/empty-dir/29897fb3-27ec-4f39-8f74-f96c5cba04a1-audit-log" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184724 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184738 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" volumeName="kubernetes.io/projected/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-kube-api-access-xg595" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184750 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184763 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-metrics-server-audit-profiles" seLinuxMountContext="" Oct 29 22:15:33.184754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184776 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-client-ca-bundle" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184789 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29897fb3-27ec-4f39-8f74-f96c5cba04a1" volumeName="kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-tls" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184804 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184817 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184839 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425b6b6b-e38b-4ceb-b473-f1c397eba5d0" volumeName="kubernetes.io/configmap/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-serviceca" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184852 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ab6261a-9f14-479b-9618-ed6672c8658d" volumeName="kubernetes.io/projected/9ab6261a-9f14-479b-9618-ed6672c8658d-kube-api-access-ttnct" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184865 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5a282-9fbb-45af-93b8-41c247e552f8" volumeName="kubernetes.io/configmap/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184879 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2768c741-ba6f-4f8f-b20a-ac431cd8b83f" volumeName="kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184892 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3189ad37-09b3-47dc-aa9e-e384d7a0ff21" volumeName="kubernetes.io/empty-dir/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-tmp-dir" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184904 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184917 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" volumeName="kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184930 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9e01536-6930-49aa-8507-203280a663f6" volumeName="kubernetes.io/projected/d9e01536-6930-49aa-8507-203280a663f6-kube-api-access-r56ll" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184941 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db3386f2-acfa-4a49-b1a2-1589a13adf52" volumeName="kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184954 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="192ace5d-ce40-41f2-b494-4dd3bfae0507" volumeName="kubernetes.io/secret/192ace5d-ce40-41f2-b494-4dd3bfae0507-agent-certs" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184966 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6844877d-8910-4d77-a31c-ca119d4f36ab" volumeName="kubernetes.io/projected/6844877d-8910-4d77-a31c-ca119d4f36ab-kube-api-access-6k2mh" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184979 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" volumeName="kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.184991 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" volumeName="kubernetes.io/empty-dir/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-textfile" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185006 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" volumeName="kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185018 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="192ace5d-ce40-41f2-b494-4dd3bfae0507" volumeName="kubernetes.io/configmap/192ace5d-ce40-41f2-b494-4dd3bfae0507-konnectivity-ca" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185031 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" volumeName="kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185044 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="690683ad-1ece-45ed-82ae-373e0a359b61" volumeName="kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-cni-binary-copy" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185056 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="697eac89-c298-46e4-ac8c-bc825959b7bd" volumeName="kubernetes.io/projected/697eac89-c298-46e4-ac8c-bc825959b7bd-kube-api-access-p7dwh" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185085 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb595d53-05aa-4575-a5e2-658dfcfbe833" volumeName="kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185099 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71169ebf-0176-4dec-8ac5-8a5adf441324" volumeName="kubernetes.io/empty-dir/71169ebf-0176-4dec-8ac5-8a5adf441324-tmp-dir" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185112 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22c03e26-3445-48ed-9296-8678cb720276" volumeName="kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-config" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185125 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="690683ad-1ece-45ed-82ae-373e0a359b61" volumeName="kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-multus-daemon-config" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185139 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71169ebf-0176-4dec-8ac5-8a5adf441324" volumeName="kubernetes.io/configmap/71169ebf-0176-4dec-8ac5-8a5adf441324-config-volume" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185151 5367 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fffeb4-7835-4126-8258-a12f509a443f" volumeName="kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-etc-tuned" seLinuxMountContext="" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185163 5367 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:15:33.187123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.185171 5367 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:15:33.211965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.211916 5367 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:15:33.213230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.213207 5367 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:15:33.213230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.213228 5367 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:15:33.213394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.213245 5367 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:15:33.213394 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:33.213280 5367 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:15:33.230998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.230965 5367 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:33.231134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.231067 5367 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:15:33.234606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.234570 5367 generic.go:356] "Generic (PLEG): container finished" podID="0dbcec17752aa0cceec127e3b277608a" containerID="308b20b95a7edd73c9162a5b02fd1e7f6c49e781e488d1ed35f825b091a01392" exitCode=0 Oct 29 22:15:33.243890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.243859 5367 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:33.246008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.245982 5367 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:33.252281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.252252 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d" exitCode=0 Oct 29 22:15:33.254557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.254533 5367 generic.go:356] "Generic (PLEG): container finished" podID="7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f" containerID="f315c38c227e158324c5e4723251f56831f0dc9b2d2511c98d168fa00f4f852b" exitCode=0 Oct 29 22:15:33.258184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.258161 5367 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:33.262612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.262587 5367 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="99c0e2fc879551ce967361e3c6727cae043d837332fc5ca9625904e74c50593d" exitCode=0 Oct 29 22:15:33.262612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.262609 5367 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="c0bf80f94d327595101f84973d222bbf06f041f4de1c7764e07139533284d228" exitCode=0 Oct 29 22:15:33.262612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.262618 5367 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="2c10c6719862f241875148a8b61b94ecb808b231fef1ad85f5e61f9313fe1a97" exitCode=0 Oct 29 22:15:33.262869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.262625 5367 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="23fa072687876d1ef4bd0a95a1ab4988a65c253600301a65764e9e9a3b2667c7" exitCode=0 Oct 29 22:15:33.262869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.262632 5367 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="05fece8901432a51753f49e6cdbbd4122f6181e46281d41db5e45a7a5b372025" exitCode=0 Oct 29 22:15:33.262869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.262639 5367 generic.go:356] "Generic (PLEG): container finished" podID="cb595d53-05aa-4575-a5e2-658dfcfbe833" containerID="9e9b9c361a270a7d0b3da8e6e956d4dd5cd7c4de1ccd1b93fc07ab571a9546f9" exitCode=0 Oct 29 22:15:33.281102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.281083 5367 manager.go:324] Recovery completed Oct 29 22:15:33.303060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.302995 5367 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:15:33.303060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303013 5367 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:15:33.303060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303032 5367 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303149 5367 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303160 5367 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303177 5367 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303182 5367 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303188 5367 policy_none.go:50] "Start" Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303196 5367 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:15:33.303237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303204 5367 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:15:33.303427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303329 5367 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:15:33.303427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303335 5367 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:15:33.303427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.303340 5367 policy_none.go:44] "Start" Oct 29 22:15:33.307781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.307744 5367 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:15:33.311379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.311365 5367 manager.go:341] "Starting Device Plugin manager" Oct 29 22:15:33.312123 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:33.311390 5367 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:15:33.312123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312116 5367 server.go:87] "Starting device plugin registration server" Oct 29 22:15:33.312453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312437 5367 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:15:33.312518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312451 5367 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:15:33.312583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312543 5367 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:15:33.312678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312650 5367 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:15:33.312749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312687 5367 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:15:33.312749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312694 5367 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:15:33.312840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.312760 5367 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:15:33.312676891Z","UUID":"c9957eae-3839-429a-81d7-3cca33312a05","Handler":null,"Name":"","Endpoint":""} Oct 29 22:15:33.313521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.313477 5367 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"] Oct 29 22:15:33.314304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314288 5367 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.314396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314322 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" event={"ID":"4ab51ab2ff14d7b6654efe4a89f4f347","Type":"ContainerStarted","Data":"468941a50a2add7ef0bdec81215abc3ac5474308bf56f5dd0f074208d334ddab"} Oct 29 22:15:33.314396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314350 5367 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:15:33.314396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314377 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" event={"ID":"4ab51ab2ff14d7b6654efe4a89f4f347","Type":"ContainerStarted","Data":"f751f947650077b1e3e9c3235972bf0b8eca67931c686624d1eeb3a837fcdc35"} Oct 29 22:15:33.314396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314386 5367 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:15:33.314609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314398 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"0dbcec17752aa0cceec127e3b277608a","Type":"ContainerStarted","Data":"d4dd3ff1e41cf2fd8127ddaa2e893529131248deb37b9f1072afc89657113ed0"} Oct 29 22:15:33.314609 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:33.314405 5367 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:15:33.314609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314419 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"0dbcec17752aa0cceec127e3b277608a","Type":"ContainerDied","Data":"308b20b95a7edd73c9162a5b02fd1e7f6c49e781e488d1ed35f825b091a01392"} Oct 29 22:15:33.314609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314444 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"0dbcec17752aa0cceec127e3b277608a","Type":"ContainerStarted","Data":"7591a580ddd1c7536f32b4202e6d1ca66aab1f92d3f8942656d8e0d1da043c00"} Oct 29 22:15:33.314809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.314674 5367 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.361263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.361233 5367 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:33.361394 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:33.361280 5367 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.413539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.413510 5367 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:15:33.415589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.415566 5367 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:15:33.415721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.415597 5367 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:15:33.415721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.415607 5367 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:15:33.415721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.415656 5367 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.431440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.431417 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.431514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.431442 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.431514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.431461 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ab51ab2ff14d7b6654efe4a89f4f347-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"4ab51ab2ff14d7b6654efe4a89f4f347\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.441843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.441824 5367 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:33.441898 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:33.441865 5367 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.451882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.451862 5367 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.451946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.451929 5367 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.532162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.532126 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.532162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.532079 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.532354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.532212 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.532354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.532232 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ab51ab2ff14d7b6654efe4a89f4f347-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"4ab51ab2ff14d7b6654efe4a89f4f347\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.532354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.532277 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ab51ab2ff14d7b6654efe4a89f4f347-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"4ab51ab2ff14d7b6654efe4a89f4f347\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" Oct 29 22:15:33.532354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:33.532297 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbcec17752aa0cceec127e3b277608a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"0dbcec17752aa0cceec127e3b277608a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:34.166501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.166450 5367 apiserver.go:52] "Watching apiserver" Oct 29 22:15:34.262292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.262262 5367 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:15:34.262839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.262818 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-x4qd2","openshift-multus/network-metrics-daemon-vmv7w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-8zr9m","openshift-network-diagnostics/network-check-target-7l9b5","kube-system/konnectivity-agent-cdbjg","openshift-dns/dns-default-7xrmb","openshift-monitoring/node-exporter-t5ml4","kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-rk9gl","openshift-cluster-node-tuning-operator/tuned-cbfz7","openshift-console/console-5bf4d96c-bcdwj","openshift-monitoring/metrics-server-684967c95f-cf5zb","openshift-multus/multus-68lrg","openshift-console/console-67b8fc9b66-29q5f","openshift-ingress-canary/ingress-canary-j8cft","openshift-network-operator/iptables-alerter-7klw5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8","openshift-dns/node-resolver-5zr4d","openshift-image-registry/node-ca-mhvkt","openshift-insights/insights-runtime-extractor-n69kw"] Oct 29 22:15:34.275408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.275379 5367 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:34.280748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280679 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:15:34.280748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280715 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:15:34.280748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280685 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.280919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280756 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w24c4\"" Oct 29 22:15:34.280919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280773 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.280919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280779 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:15:34.280919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280877 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:15:34.280919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280897 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:15:34.281056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280940 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hznbv\"" Oct 29 22:15:34.281056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.280975 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:15:34.282034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.282014 5367 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:15:34.292492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292468 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.292591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292545 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.292591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292561 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:15:34.292591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292564 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.292591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292584 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:15:34.292809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292592 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:15:34.292809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292640 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.292809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.292727 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.293749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.293736 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.295489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295471 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:15:34.295575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295560 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.295637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295628 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-475gp\"" Oct 29 22:15:34.295725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295709 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:15:34.295788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295767 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.295788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295783 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:15:34.295882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295788 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.295882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295805 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-c2grw\"" Oct 29 22:15:34.295882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295835 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:15:34.295882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.295869 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kk6hd\"" Oct 29 22:15:34.296066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.296050 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:15:34.296164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.296151 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.296310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.296295 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:15:34.307849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.307828 5367 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:34.307924 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:34.307875 5367 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" Oct 29 22:15:34.307924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.307881 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:15:34.337523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337504 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-netd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.337614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337529 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.337614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337553 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.337614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337570 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-netns\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.337770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337622 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-bin\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.337770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337671 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.337770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337687 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.337770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337733 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/697eac89-c298-46e4-ac8c-bc825959b7bd-host-slash\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.337770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337759 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.337770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337759 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337817 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337860 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-tmp\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337900 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337932 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-device-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337957 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-cnibin\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337980 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-tmp\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.337984 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-var-lib-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338009 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-systemd\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.338047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338035 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338058 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-serviceca\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338094 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-conf-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338135 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmftn\" (UniqueName: \"kubernetes.io/projected/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-kube-api-access-nmftn\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338159 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-metrics-client-ca\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338215 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338302 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dwh\" (UniqueName: \"kubernetes.io/projected/697eac89-c298-46e4-ac8c-bc825959b7bd-kube-api-access-p7dwh\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338334 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-kubernetes\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338368 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-k8s-cni-cncf-io\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338394 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.338434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338424 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b5a282-9fbb-45af-93b8-41c247e552f8-data-volume\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338493 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-kubelet\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338532 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-slash\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338533 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9b5a282-9fbb-45af-93b8-41c247e552f8-data-volume\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338553 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtl7r\" (UniqueName: \"kubernetes.io/projected/83fffeb4-7835-4126-8258-a12f509a443f-kube-api-access-jtl7r\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338575 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56ll\" (UniqueName: \"kubernetes.io/projected/d9e01536-6930-49aa-8507-203280a663f6-kube-api-access-r56ll\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338590 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmzt\" (UniqueName: \"kubernetes.io/projected/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-kube-api-access-lmmzt\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338628 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-serviceca\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338626 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-ovn\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338668 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338693 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6844877d-8910-4d77-a31c-ca119d4f36ab-cert\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338805 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-script-lib\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.338908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338892 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338934 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338961 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6844877d-8910-4d77-a31c-ca119d4f36ab-cert\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.338971 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339043 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-script-lib\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339041 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-hosts-file\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339107 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-systemd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339125 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-bin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339148 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339156 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339206 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-client-certs\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339306 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339336 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339369 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-multus-daemon-config\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339395 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5rh\" (UniqueName: \"kubernetes.io/projected/cb595d53-05aa-4575-a5e2-658dfcfbe833-kube-api-access-6r5rh\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339420 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-cnibin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339444 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71169ebf-0176-4dec-8ac5-8a5adf441324-metrics-tls\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.339469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339468 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71169ebf-0176-4dec-8ac5-8a5adf441324-tmp-dir\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339622 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71169ebf-0176-4dec-8ac5-8a5adf441324-tmp-dir\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339633 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339681 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339723 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-root\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339756 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339784 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339815 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339860 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339887 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-hostroot\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339919 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg595\" (UniqueName: \"kubernetes.io/projected/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-kube-api-access-xg595\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339943 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-etc-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339967 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.339992 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340016 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-system-cni-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340041 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-tls\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340068 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340073 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340094 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-original-pull-secret\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340176 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-systemd-units\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340210 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysconfig\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.340255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340236 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn97\" (UniqueName: \"kubernetes.io/projected/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-api-access-lcn97\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340314 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-log-socket\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340340 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m27b5\" (UniqueName: \"kubernetes.io/projected/71169ebf-0176-4dec-8ac5-8a5adf441324-kube-api-access-m27b5\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340305 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340375 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340491 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-multus\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340525 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-wtmp\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340546 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/697eac89-c298-46e4-ac8c-bc825959b7bd-iptables-alerter-script\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340569 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-lib-modules\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340628 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340651 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5a282-9fbb-45af-93b8-41c247e552f8-crio-socket\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340665 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-dbus\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340681 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340714 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340739 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/697eac89-c298-46e4-ac8c-bc825959b7bd-iptables-alerter-script\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340800 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bkf\" (UniqueName: \"kubernetes.io/projected/29897fb3-27ec-4f39-8f74-f96c5cba04a1-kube-api-access-g2bkf\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340827 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340855 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340889 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71169ebf-0176-4dec-8ac5-8a5adf441324-config-volume\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340914 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-sys-fs\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340932 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340939 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-host\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.340959 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341004 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-registration-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.341052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341035 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/192ace5d-ce40-41f2-b494-4dd3bfae0507-konnectivity-ca\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341071 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-metrics-server-audit-profiles\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341074 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71169ebf-0176-4dec-8ac5-8a5adf441324-config-volume\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341091 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-config\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341111 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c03e26-3445-48ed-9296-8678cb720276-ovn-node-metrics-cert\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341143 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341179 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341207 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-multus-certs\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341231 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-etc-tuned\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341258 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnct\" (UniqueName: \"kubernetes.io/projected/9ab6261a-9f14-479b-9618-ed6672c8658d-kube-api-access-ttnct\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341298 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-ovnkube-config\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341302 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-socket-dir-parent\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341325 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83fffeb4-7835-4126-8258-a12f509a443f-etc-tuned\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341355 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb595d53-05aa-4575-a5e2-658dfcfbe833-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341387 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c03e26-3445-48ed-9296-8678cb720276-ovn-node-metrics-cert\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341446 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341466 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-modprobe-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341482 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-var-lib-kubelet\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341515 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341539 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/192ace5d-ce40-41f2-b494-4dd3bfae0507-agent-certs\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341573 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-textfile\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341614 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-tls\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341643 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-kubelet-config\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341655 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-textfile\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341674 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-env-overrides\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341728 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-socket-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341758 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341794 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9b5a282-9fbb-45af-93b8-41c247e552f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341813 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c03e26-3445-48ed-9296-8678cb720276-env-overrides\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341820 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-host\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341873 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-cni-binary-copy\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341951 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-client-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341975 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-node-log\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341991 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342006 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-cni-binary-copy\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342015 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbw4\" (UniqueName: \"kubernetes.io/projected/690683ad-1ece-45ed-82ae-373e0a359b61-kube-api-access-lfbw4\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.341990 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9b5a282-9fbb-45af-93b8-41c247e552f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342041 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/29897fb3-27ec-4f39-8f74-f96c5cba04a1-audit-log\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342060 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-conf\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342074 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-sys\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342091 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2mh\" (UniqueName: \"kubernetes.io/projected/6844877d-8910-4d77-a31c-ca119d4f36ab-kube-api-access-6k2mh\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342131 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/29897fb3-27ec-4f39-8f74-f96c5cba04a1-audit-log\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342141 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342178 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-kubelet\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342207 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqvh\" (UniqueName: \"kubernetes.io/projected/22c03e26-3445-48ed-9296-8678cb720276-kube-api-access-hhqvh\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342233 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342266 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342297 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342319 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342341 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-etc-kubernetes\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342418 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-sys\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342469 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-netns\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342494 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-run\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342520 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342541 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342562 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-system-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342592 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-tmp-dir\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342656 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342685 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-tmp-dir\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.343168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.342760 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.358237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.358211 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmzt\" (UniqueName: \"kubernetes.io/projected/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-kube-api-access-lmmzt\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.358390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.358375 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt97c\" (UniqueName: \"kubernetes.io/projected/12e17b5f-8baa-4708-8bef-2660e45d479c-kube-api-access-lt97c\") pod \"network-check-target-7l9b5\" (UID: \"12e17b5f-8baa-4708-8bef-2660e45d479c\") " pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:34.362658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.362640 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqvh\" (UniqueName: \"kubernetes.io/projected/22c03e26-3445-48ed-9296-8678cb720276-kube-api-access-hhqvh\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.381408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.381386 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.381408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.381400 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 29 22:15:34.381688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.381662 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.381851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.381729 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:15:34.381851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.381758 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.382034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.381865 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.382034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382010 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 29 22:15:34.382034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382010 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:15:34.382370 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382048 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:15:34.382370 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382148 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.382552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382401 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382631 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxx6w\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382669 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382767 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382777 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382788 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382803 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382971 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:15:34.382920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.382993 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383030 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383045 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383216 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383327 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383440 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383461 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383543 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bkf\" (UniqueName: \"kubernetes.io/projected/29897fb3-27ec-4f39-8f74-f96c5cba04a1-kube-api-access-g2bkf\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383611 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-657mjem3c3csp\"" Oct 29 22:15:34.383723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.383668 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbw4\" (UniqueName: \"kubernetes.io/projected/690683ad-1ece-45ed-82ae-373e0a359b61-kube-api-access-lfbw4\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.384053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384030 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.384223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384203 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h6g72\"" Oct 29 22:15:34.384256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384209 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2mh\" (UniqueName: \"kubernetes.io/projected/6844877d-8910-4d77-a31c-ca119d4f36ab-kube-api-access-6k2mh\") pod \"ingress-canary-j8cft\" (UID: \"6844877d-8910-4d77-a31c-ca119d4f36ab\") " pod="openshift-ingress-canary/ingress-canary-j8cft" Oct 29 22:15:34.384434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384396 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:15:34.384728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384660 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbljw\"" Oct 29 22:15:34.384778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384712 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:15:34.384778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384769 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.384834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384771 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:15:34.384834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384786 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 22:15:34.384834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384813 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:15:34.384960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.384939 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2cw6s\"" Oct 29 22:15:34.385042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385024 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 29 22:15:34.385096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385067 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:15:34.385144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385126 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:15:34.385189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385168 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:15:34.386121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385840 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm48d\"" Oct 29 22:15:34.386121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385935 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 22:15:34.386121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.385959 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rfsln\"" Oct 29 22:15:34.386121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386006 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9x6m\"" Oct 29 22:15:34.386121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386063 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkqw\"" Oct 29 22:15:34.386453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386193 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:15:34.386453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386371 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:15:34.386594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386575 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:15:34.386737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386718 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:15:34.386884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.386859 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:15:34.387518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387018 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:15:34.387518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387047 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:15:34.387518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387134 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28glh\"" Oct 29 22:15:34.387518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387299 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qnnrl\"" Oct 29 22:15:34.387518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387473 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:15:34.387781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387623 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-thfrk\"" Oct 29 22:15:34.387838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387815 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmftn\" (UniqueName: \"kubernetes.io/projected/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-kube-api-access-nmftn\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.387970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387933 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:15:34.387970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.387954 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.388320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388302 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.388518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388501 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.388572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388500 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtl7r\" (UniqueName: \"kubernetes.io/projected/83fffeb4-7835-4126-8258-a12f509a443f-kube-api-access-jtl7r\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.388681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388661 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.388762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388678 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-metrics-client-ca\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.388762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388677 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.388950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.388913 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.389168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389133 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.389270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389250 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56ll\" (UniqueName: \"kubernetes.io/projected/d9e01536-6930-49aa-8507-203280a663f6-kube-api-access-r56ll\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:34.389325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389249 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dwh\" (UniqueName: \"kubernetes.io/projected/697eac89-c298-46e4-ac8c-bc825959b7bd-kube-api-access-p7dwh\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.389438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389420 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 29 22:15:34.389500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389469 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-client-certs\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.389660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389645 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.389660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389652 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/690683ad-1ece-45ed-82ae-373e0a359b61-multus-daemon-config\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.389851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389832 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.389944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.389899 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71169ebf-0176-4dec-8ac5-8a5adf441324-metrics-tls\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.390038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.390021 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.390197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.390182 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.390385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.390367 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-tls\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.391105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391082 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.391242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391222 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.391301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391258 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.391301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391271 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/192ace5d-ce40-41f2-b494-4dd3bfae0507-konnectivity-ca\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:15:34.391381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391329 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.391512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391494 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.391593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391508 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/29897fb3-27ec-4f39-8f74-f96c5cba04a1-metrics-server-audit-profiles\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.391649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391601 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg595\" (UniqueName: \"kubernetes.io/projected/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-kube-api-access-xg595\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.391752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391730 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.391817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391730 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5rh\" (UniqueName: \"kubernetes.io/projected/cb595d53-05aa-4575-a5e2-658dfcfbe833-kube-api-access-6r5rh\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.391817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391766 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/192ace5d-ce40-41f2-b494-4dd3bfae0507-agent-certs\") pod \"konnectivity-agent-cdbjg\" (UID: \"192ace5d-ce40-41f2-b494-4dd3bfae0507\") " pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:15:34.391817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.391911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.391878 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-secret-metrics-server-tls\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.392018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392006 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e01536-6930-49aa-8507-203280a663f6-metrics-certs\") pod \"network-metrics-daemon-vmv7w\" (UID: \"d9e01536-6930-49aa-8507-203280a663f6\") " pod="openshift-multus/network-metrics-daemon-vmv7w" Oct 29 22:15:34.392150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392136 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/29897fb3-27ec-4f39-8f74-f96c5cba04a1-client-ca-bundle\") pod \"metrics-server-684967c95f-cf5zb\" (UID: \"29897fb3-27ec-4f39-8f74-f96c5cba04a1\") " pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:34.392414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392399 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m27b5\" (UniqueName: \"kubernetes.io/projected/71169ebf-0176-4dec-8ac5-8a5adf441324-kube-api-access-m27b5\") pod \"dns-default-7xrmb\" (UID: \"71169ebf-0176-4dec-8ac5-8a5adf441324\") " pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:34.392563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392548 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.392728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392679 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") pod \"console-5bf4d96c-bcdwj\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:34.392879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392858 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnct\" (UniqueName: \"kubernetes.io/projected/9ab6261a-9f14-479b-9618-ed6672c8658d-kube-api-access-ttnct\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.392921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392876 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn97\" (UniqueName: \"kubernetes.io/projected/a9b5a282-9fbb-45af-93b8-41c247e552f8-kube-api-access-lcn97\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.392921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392892 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:34.392921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.392905 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.398247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.398232 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") pod \"console-67b8fc9b66-29q5f\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:34.443361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443308 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-cnibin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443332 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443347 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-root\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443362 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443379 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-hostroot\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443425 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443435 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443445 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-root\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443447 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-cnibin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443471 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-etc-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443451 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-etc-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443480 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-hostroot\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443506 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443525 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-system-cni-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443541 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-systemd-units\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443556 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysconfig\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443572 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443580 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-log-socket\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443593 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-systemd-units\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443596 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-system-cni-dir\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443612 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-log-socket\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443614 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-multus\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443630 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysconfig\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443646 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-multus\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443671 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-wtmp\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443691 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-lib-modules\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443727 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5a282-9fbb-45af-93b8-41c247e552f8-crio-socket\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443742 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-dbus\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443773 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-node-exporter-wtmp\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443793 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-sys-fs\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443798 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-dbus\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443796 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5a282-9fbb-45af-93b8-41c247e552f8-crio-socket\") pod \"insights-runtime-extractor-n69kw\" (UID: \"a9b5a282-9fbb-45af-93b8-41c247e552f8\") " pod="openshift-insights/insights-runtime-extractor-n69kw" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443811 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443814 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-host\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443846 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-lib-modules\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.443853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443854 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-host\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443861 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-registration-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443889 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-sys-fs\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443898 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-registration-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443934 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-multus-certs\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443949 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-socket-dir-parent\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443958 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-multus-certs\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443964 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-modprobe-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443978 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-var-lib-kubelet\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.443993 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-kubelet-config\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444011 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-socket-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444047 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-socket-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444048 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-var-lib-kubelet\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444058 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-socket-dir-parent\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444068 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-host\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444090 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-node-log\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444106 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-conf\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444120 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-sys\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444071 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2768c741-ba6f-4f8f-b20a-ac431cd8b83f-kubelet-config\") pod \"global-pull-secret-syncer-rk9gl\" (UID: \"2768c741-ba6f-4f8f-b20a-ac431cd8b83f\") " pod="kube-system/global-pull-secret-syncer-rk9gl" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444129 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425b6b6b-e38b-4ceb-b473-f1c397eba5d0-host\") pod \"node-ca-mhvkt\" (UID: \"425b6b6b-e38b-4ceb-b473-f1c397eba5d0\") " pod="openshift-image-registry/node-ca-mhvkt" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444066 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-modprobe-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444151 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-kubelet\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444163 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-sys\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444190 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-node-log\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444220 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-kubelet\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444222 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-conf\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444245 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-etc-kubernetes\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444267 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-sys\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444281 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-netns\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444294 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-run\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444308 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-system-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444337 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-netns\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444344 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-etc-kubernetes\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444354 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-netd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444368 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-run\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444323 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f-sys\") pod \"node-exporter-t5ml4\" (UID: \"7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f\") " pod="openshift-monitoring/node-exporter-t5ml4" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444394 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-netd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444379 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-netns\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444388 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-system-cni-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444427 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-netns\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444452 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-bin\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444473 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/697eac89-c298-46e4-ac8c-bc825959b7bd-host-slash\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444494 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/697eac89-c298-46e4-ac8c-bc825959b7bd-host-slash\") pod \"iptables-alerter-7klw5\" (UID: \"697eac89-c298-46e4-ac8c-bc825959b7bd\") " pod="openshift-network-operator/iptables-alerter-7klw5" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444512 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-cni-bin\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444537 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444560 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444580 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-device-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444598 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-cnibin\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.444588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444613 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-var-lib-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444627 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-systemd\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444641 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-conf-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444660 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-kubernetes\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444678 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb595d53-05aa-4575-a5e2-658dfcfbe833-cnibin\") pod \"multus-additional-cni-plugins-x4qd2\" (UID: \"cb595d53-05aa-4575-a5e2-658dfcfbe833\") " pod="openshift-multus/multus-additional-cni-plugins-x4qd2" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444683 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-k8s-cni-cncf-io\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444687 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-var-lib-openvswitch\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444726 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-systemd\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444732 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-kubelet\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444737 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-kubernetes\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444743 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-device-dir\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444728 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-multus-conf-dir\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444764 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-slash\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444788 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-kubelet\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444792 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-ovn\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444781 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-run-k8s-cni-cncf-io\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444820 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444827 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-slash\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444835 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-ovn\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444877 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-hosts-file\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444891 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83fffeb4-7835-4126-8258-a12f509a443f-etc-sysctl-d\") pod \"tuned-cbfz7\" (UID: \"83fffeb4-7835-4126-8258-a12f509a443f\") " pod="openshift-cluster-node-tuning-operator/tuned-cbfz7" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444904 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-systemd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444930 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-bin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444954 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444959 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-run-systemd\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444963 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690683ad-1ece-45ed-82ae-373e0a359b61-host-var-lib-cni-bin\") pod \"multus-68lrg\" (UID: \"690683ad-1ece-45ed-82ae-373e0a359b61\") " pod="openshift-multus/multus-68lrg" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444970 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3189ad37-09b3-47dc-aa9e-e384d7a0ff21-hosts-file\") pod \"node-resolver-5zr4d\" (UID: \"3189ad37-09b3-47dc-aa9e-e384d7a0ff21\") " pod="openshift-dns/node-resolver-5zr4d" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444981 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.444990 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c03e26-3445-48ed-9296-8678cb720276-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zr9m\" (UID: \"22c03e26-3445-48ed-9296-8678cb720276\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:34.446034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.445014 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ab6261a-9f14-479b-9618-ed6672c8658d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlnj8\" (UID: \"9ab6261a-9f14-479b-9618-ed6672c8658d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlnj8" Oct 29 22:15:34.583796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:34.583768 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7xrmb" Oct 29 22:15:35.592272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:35.592243 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:36.298841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:36.298810 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:36.299015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:36.299003 5367 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:15:36.729633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:36.729601 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:36.734476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:36.734453 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:40.642064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:40.642015 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:41.114869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:41.114832 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:15:49.973859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:49.973781 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" Oct 29 22:15:50.647257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:50.647229 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:15:52.329192 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:52.329159 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:15:53.035314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:53.035266 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cdbjg" Oct 29 22:15:54.307101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:54.307068 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zr9m" Oct 29 22:15:55.351249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351213 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:55.351892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351570 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="alertmanager" containerID="cri-o://6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71" gracePeriod=120 Oct 29 22:15:55.351892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351633 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-web" containerID="cri-o://9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef" gracePeriod=120 Oct 29 22:15:55.351892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351626 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-metric" containerID="cri-o://5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3" gracePeriod=120 Oct 29 22:15:55.351892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351656 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="config-reloader" containerID="cri-o://c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618" gracePeriod=120 Oct 29 22:15:55.351892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351686 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy" containerID="cri-o://74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f" gracePeriod=120 Oct 29 22:15:55.351892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:55.351715 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="prom-label-proxy" containerID="cri-o://02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a" gracePeriod=120 Oct 29 22:15:56.352579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352546 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a" exitCode=0 Oct 29 22:15:56.352579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352570 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3" exitCode=0 Oct 29 22:15:56.352579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352583 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f" exitCode=0 Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352591 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618" exitCode=0 Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352599 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71" exitCode=0 Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352614 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a"} Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352707 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3"} Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352734 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f"} Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352759 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618"} Oct 29 22:15:56.353025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.352783 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71"} Oct 29 22:15:56.579482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.579456 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:56.695272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695196 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695233 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695260 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695282 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695313 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695350 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695661 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695743 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695758 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:56.695793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695777 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695813 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695866 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.695941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695913 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.696099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.695976 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") pod \"db3386f2-acfa-4a49-b1a2-1589a13adf52\" (UID: \"db3386f2-acfa-4a49-b1a2-1589a13adf52\") " Oct 29 22:15:56.696233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.696211 5367 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.696746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.696691 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:56.697832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.697802 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:56.699595 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.699569 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:56.699730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.699567 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "kube-api-access-94z6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:56.700058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.700026 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.700361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.700316 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.700457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.700361 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:56.700457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.700385 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.700655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.700631 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.701489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.701467 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.703880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.703853 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.708487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.708465 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config" pod "db3386f2-acfa-4a49-b1a2-1589a13adf52" (UID: "db3386f2-acfa-4a49-b1a2-1589a13adf52"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:56.796757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796719 5367 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-cluster-tls-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.796757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796751 5367 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-main-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.796757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796764 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94z6s\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-kube-api-access-94z6s\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796776 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3386f2-acfa-4a49-b1a2-1589a13adf52-metrics-client-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796788 5367 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-out\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796804 5367 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796815 5367 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3386f2-acfa-4a49-b1a2-1589a13adf52-tls-assets\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796827 5367 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796838 5367 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796850 5367 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796862 5367 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3386f2-acfa-4a49-b1a2-1589a13adf52-alertmanager-main-db\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.797027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:56.796874 5367 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3386f2-acfa-4a49-b1a2-1589a13adf52-web-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:57.359072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.359035 5367 generic.go:356] "Generic (PLEG): container finished" podID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerID="9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef" exitCode=0 Oct 29 22:15:57.359491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.359088 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef"} Oct 29 22:15:57.359491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.359113 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.359491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.359158 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3386f2-acfa-4a49-b1a2-1589a13adf52","Type":"ContainerDied","Data":"450512304ed638d6cf7ec08867e405b2ea6baea4eec6641211ab6e13b123d451"} Oct 29 22:15:57.359491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.359191 5367 scope.go:122] "RemoveContainer" containerID="02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a" Oct 29 22:15:57.366560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.366544 5367 scope.go:122] "RemoveContainer" containerID="5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3" Oct 29 22:15:57.372646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.372632 5367 scope.go:122] "RemoveContainer" containerID="74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f" Oct 29 22:15:57.378874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.378861 5367 scope.go:122] "RemoveContainer" containerID="9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef" Oct 29 22:15:57.384801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.384780 5367 scope.go:122] "RemoveContainer" containerID="c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618" Oct 29 22:15:57.392299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.392279 5367 scope.go:122] "RemoveContainer" containerID="6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71" Oct 29 22:15:57.398983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.398968 5367 scope.go:122] "RemoveContainer" containerID="335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d" Oct 29 22:15:57.405252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405238 5367 scope.go:122] "RemoveContainer" containerID="02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a" Oct 29 22:15:57.405489 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.405469 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a\": container with ID starting with 02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a not found: ID does not exist" containerID="02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a" Oct 29 22:15:57.405537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405498 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a"} err="failed to get container status \"02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a\": rpc error: code = NotFound desc = could not find container \"02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a\": container with ID starting with 02f818357347ad90f1b1ddc8cbf0d4f06b962c8e234bb0f540c32f3f51c33a7a not found: ID does not exist" Oct 29 22:15:57.405537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405529 5367 scope.go:122] "RemoveContainer" containerID="5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3" Oct 29 22:15:57.405753 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.405737 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3\": container with ID starting with 5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3 not found: ID does not exist" containerID="5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3" Oct 29 22:15:57.405803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405759 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3"} err="failed to get container status \"5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3\": rpc error: code = NotFound desc = could not find container \"5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3\": container with ID starting with 5cec19f4f8706b9aaa9193b33378556e5007090be5c57a937cd59234b54b41a3 not found: ID does not exist" Oct 29 22:15:57.405803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405773 5367 scope.go:122] "RemoveContainer" containerID="74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f" Oct 29 22:15:57.405964 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.405947 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f\": container with ID starting with 74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f not found: ID does not exist" containerID="74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f" Oct 29 22:15:57.405998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405968 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f"} err="failed to get container status \"74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f\": rpc error: code = NotFound desc = could not find container \"74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f\": container with ID starting with 74352df91a0a46e460f03897cf1033712ecc6152aa537cf0196cb52a0d4fb54f not found: ID does not exist" Oct 29 22:15:57.405998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.405980 5367 scope.go:122] "RemoveContainer" containerID="9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef" Oct 29 22:15:57.406189 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.406172 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef\": container with ID starting with 9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef not found: ID does not exist" containerID="9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef" Oct 29 22:15:57.406250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.406199 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef"} err="failed to get container status \"9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef\": rpc error: code = NotFound desc = could not find container \"9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef\": container with ID starting with 9f9cec6bf10cd4b522f0590e921b29fd92cc52056c15590a37c3c105f6adafef not found: ID does not exist" Oct 29 22:15:57.406250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.406223 5367 scope.go:122] "RemoveContainer" containerID="c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618" Oct 29 22:15:57.406466 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.406449 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618\": container with ID starting with c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618 not found: ID does not exist" containerID="c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618" Oct 29 22:15:57.406523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.406471 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618"} err="failed to get container status \"c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618\": rpc error: code = NotFound desc = could not find container \"c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618\": container with ID starting with c781751d81cc38127ade741f8309c7dd2242b55a50e6f80656daaba0229ed618 not found: ID does not exist" Oct 29 22:15:57.406523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.406486 5367 scope.go:122] "RemoveContainer" containerID="6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71" Oct 29 22:15:57.406693 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.406676 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71\": container with ID starting with 6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71 not found: ID does not exist" containerID="6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71" Oct 29 22:15:57.406841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.406822 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71"} err="failed to get container status \"6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71\": rpc error: code = NotFound desc = could not find container \"6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71\": container with ID starting with 6521483b8dba930e5a0cc1a0ce1bee72a31153563ee47974c3b53bebc3647d71 not found: ID does not exist" Oct 29 22:15:57.406907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.406843 5367 scope.go:122] "RemoveContainer" containerID="335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d" Oct 29 22:15:57.407129 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:15:57.407113 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d\": container with ID starting with 335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d not found: ID does not exist" containerID="335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d" Oct 29 22:15:57.407184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.407135 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d"} err="failed to get container status \"335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d\": rpc error: code = NotFound desc = could not find container \"335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d\": container with ID starting with 335c6477e77b7ce1601b6c5854745baf48fb997afdf836074d344a1c2c27539d not found: ID does not exist" Oct 29 22:15:57.429996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.429977 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:57.444111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.444088 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:57.462809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.462785 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:57.462997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.462986 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-web" Oct 29 22:15:57.463037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.462998 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-web" Oct 29 22:15:57.463037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463008 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="alertmanager" Oct 29 22:15:57.463037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463013 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="alertmanager" Oct 29 22:15:57.463037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463021 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-metric" Oct 29 22:15:57.463037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463026 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-metric" Oct 29 22:15:57.463037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463037 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="init-config-reloader" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463042 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="init-config-reloader" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463052 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463056 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463066 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="prom-label-proxy" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463071 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="prom-label-proxy" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463078 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="config-reloader" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463082 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="config-reloader" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463120 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="init-config-reloader" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463128 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463133 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="prom-label-proxy" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463139 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="config-reloader" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463145 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-web" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463149 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="alertmanager" Oct 29 22:15:57.463198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.463154 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" containerName="kube-rbac-proxy-metric" Oct 29 22:15:57.466458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.466442 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.483002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.482949 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:15:57.483151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483120 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-thfrk\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483158 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483166 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483123 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483171 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483122 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483197 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:15:57.483262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.483179 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:15:57.497246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.497229 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:15:57.500597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.500578 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:57.502663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502642 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592888da-77f8-4787-ad63-8d4a1a2a4b31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.502781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502686 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/592888da-77f8-4787-ad63-8d4a1a2a4b31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.502781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502727 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.502894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502817 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/592888da-77f8-4787-ad63-8d4a1a2a4b31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.502894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502881 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-config-volume\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.502996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502910 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.502996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.502940 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.503068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.503053 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.503113 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.503079 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/592888da-77f8-4787-ad63-8d4a1a2a4b31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.503166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.503149 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-web-config\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.503222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.503175 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.503222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.503194 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25bb\" (UniqueName: \"kubernetes.io/projected/592888da-77f8-4787-ad63-8d4a1a2a4b31-kube-api-access-r25bb\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.503222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.503218 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/592888da-77f8-4787-ad63-8d4a1a2a4b31-config-out\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604024 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604054 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/592888da-77f8-4787-ad63-8d4a1a2a4b31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604077 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-web-config\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604191 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604234 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25bb\" (UniqueName: \"kubernetes.io/projected/592888da-77f8-4787-ad63-8d4a1a2a4b31-kube-api-access-r25bb\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604272 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/592888da-77f8-4787-ad63-8d4a1a2a4b31-config-out\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604311 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592888da-77f8-4787-ad63-8d4a1a2a4b31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604355 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/592888da-77f8-4787-ad63-8d4a1a2a4b31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604381 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604410 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/592888da-77f8-4787-ad63-8d4a1a2a4b31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604380 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/592888da-77f8-4787-ad63-8d4a1a2a4b31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604444 5367 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:15:57.604598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604441 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-config-volume\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604489 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.604598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.604518 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.605887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.605818 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592888da-77f8-4787-ad63-8d4a1a2a4b31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.605887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.605851 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/592888da-77f8-4787-ad63-8d4a1a2a4b31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.607142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.607116 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/592888da-77f8-4787-ad63-8d4a1a2a4b31-config-out\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.607632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.607291 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.607632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.607389 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-config-volume\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.607632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.607594 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.607865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.607748 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.607958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.607933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.608320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.608303 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/592888da-77f8-4787-ad63-8d4a1a2a4b31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.608543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.608525 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.608990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.608975 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/592888da-77f8-4787-ad63-8d4a1a2a4b31-web-config\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.624071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.624052 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25bb\" (UniqueName: \"kubernetes.io/projected/592888da-77f8-4787-ad63-8d4a1a2a4b31-kube-api-access-r25bb\") pod \"alertmanager-main-0\" (UID: \"592888da-77f8-4787-ad63-8d4a1a2a4b31\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:57.783041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:57.782945 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:15:58.099477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:58.099446 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:15:58.102521 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:58.102485 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592888da_77f8_4787_ad63_8d4a1a2a4b31.slice/crio-dcc4ef6769431f7b6c5085a02cb5e5eca61a07df2fa988cea2bb19e46af436f6 WatchSource:0}: Error finding container dcc4ef6769431f7b6c5085a02cb5e5eca61a07df2fa988cea2bb19e46af436f6: Status 404 returned error can't find the container with id dcc4ef6769431f7b6c5085a02cb5e5eca61a07df2fa988cea2bb19e46af436f6 Oct 29 22:15:58.364989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:58.364909 5367 generic.go:356] "Generic (PLEG): container finished" podID="592888da-77f8-4787-ad63-8d4a1a2a4b31" containerID="c5cc2c150444ae9918eb99c0f1d677c8a3c3124897e75d18eb4f6be93cfb3f49" exitCode=0 Oct 29 22:15:58.364989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:58.364952 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerDied","Data":"c5cc2c150444ae9918eb99c0f1d677c8a3c3124897e75d18eb4f6be93cfb3f49"} Oct 29 22:15:58.365406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:58.365314 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"dcc4ef6769431f7b6c5085a02cb5e5eca61a07df2fa988cea2bb19e46af436f6"} Oct 29 22:15:59.219001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.218967 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3386f2-acfa-4a49-b1a2-1589a13adf52" path="/var/lib/kubelet/pods/db3386f2-acfa-4a49-b1a2-1589a13adf52/volumes" Oct 29 22:15:59.294455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.294422 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5bf4d96c-bcdwj" podUID="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" containerName="console" containerID="cri-o://47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a" gracePeriod=15 Oct 29 22:15:59.305616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.305593 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-767dbcc698-ncbkm"] Oct 29 22:15:59.307622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.307606 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320456 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-federate-client-tls\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320489 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320523 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxnxp\" (UniqueName: \"kubernetes.io/projected/23715cbc-f9cd-496d-9e45-c2306e7aa75c-kube-api-access-pxnxp\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320625 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-serving-certs-ca-bundle\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320707 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320655 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320732 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-secret-telemeter-client\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320747 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-metrics-client-ca\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.320775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.320769 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-telemeter-client-tls\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.323674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.323626 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 29 22:15:59.324132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.324115 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 29 22:15:59.324207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.324169 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 29 22:15:59.324256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.324209 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 29 22:15:59.338070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.338021 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 29 22:15:59.338070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.338044 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bw8xt\"" Oct 29 22:15:59.352264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.352212 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 29 22:15:59.353285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.353265 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-767dbcc698-ncbkm"] Oct 29 22:15:59.371634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.371611 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"08a3af73fd9a0862fbde546d430b440e9c7275a7cf1f194c0892c44a5986b631"} Oct 29 22:15:59.371946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.371665 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"0e01ace9e011c417a20fe71783cdf00a6de0912556c36c1a12eb36b0e5dd5535"} Oct 29 22:15:59.371946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.371692 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"458c2772621ed646bf43e0abb727f6e3273a8847c17f91fe3ea6797b17186c8d"} Oct 29 22:15:59.371946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.371737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"a574cab5a491e2ce77d38576493766945eeaffc28e0d885fa732b2ebc16c826b"} Oct 29 22:15:59.371946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.371761 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"95d6578cfa470efb1372186bf8c0b657ad00cc73760874f1774d627b48b93485"} Oct 29 22:15:59.371946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.371784 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"592888da-77f8-4787-ad63-8d4a1a2a4b31","Type":"ContainerStarted","Data":"996cd6185c654bb3e2896857efb87901db35a1861268b0df466df182a281e139"} Oct 29 22:15:59.421088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.420947 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.408998002 podStartE2EDuration="2.420930315s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.932313ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:59.419538262 +0000 UTC m=+26.313613184" watchObservedRunningTime="2026-10-29 22:15:59.420930315 +0000 UTC m=+26.315005239" Oct 29 22:15:59.421592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.421573 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxnxp\" (UniqueName: \"kubernetes.io/projected/23715cbc-f9cd-496d-9e45-c2306e7aa75c-kube-api-access-pxnxp\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.421672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.421619 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-serving-certs-ca-bundle\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.421672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.421649 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.422045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.422018 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-secret-telemeter-client\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.422346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.422059 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-metrics-client-ca\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.422346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.422115 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-telemeter-client-tls\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.422346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.422203 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-federate-client-tls\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.422346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.422233 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.423478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.422461 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-serving-certs-ca-bundle\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.423478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.423007 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-metrics-client-ca\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.423478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.423446 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23715cbc-f9cd-496d-9e45-c2306e7aa75c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.424323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.424302 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.425072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.425050 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-federate-client-tls\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.425142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.425105 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-secret-telemeter-client\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.425425 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.425404 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23715cbc-f9cd-496d-9e45-c2306e7aa75c-telemeter-client-tls\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.441968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.441937 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxnxp\" (UniqueName: \"kubernetes.io/projected/23715cbc-f9cd-496d-9e45-c2306e7aa75c-kube-api-access-pxnxp\") pod \"telemeter-client-767dbcc698-ncbkm\" (UID: \"23715cbc-f9cd-496d-9e45-c2306e7aa75c\") " pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.519518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.519496 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:15:59.624163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624134 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624172 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624217 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624266 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624297 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624338 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624443 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:59.624622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624447 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624572 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") pod \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\" (UID: \"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae\") " Oct 29 22:15:59.624737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624568 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.624737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624613 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.624844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624810 5367 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.624844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624830 5367 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-trusted-ca-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.624844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624830 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.624989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624841 5367 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-tmp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.625038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.624994 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.626858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.626837 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:59.627068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.627046 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:59.627110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.627064 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz" pod "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" (UID: "3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae"). InnerVolumeSpecName "kube-api-access-lcrdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:59.670404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.670380 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" Oct 29 22:15:59.730092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.726560 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrdz\" (UniqueName: \"kubernetes.io/projected/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-kube-api-access-lcrdz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.730092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.726609 5367 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.730092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.726624 5367 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-serving-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.730092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.726639 5367 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-oauth-serving-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.730092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.726659 5367 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae-console-oauth-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.973254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.973218 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-767dbcc698-ncbkm"] Oct 29 22:15:59.976096 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:15:59.976065 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23715cbc_f9cd_496d_9e45_c2306e7aa75c.slice/crio-416cc9f73dbe45fabc3a3574f465bee3805bd773442b2d467822bb826f76da2e WatchSource:0}: Error finding container 416cc9f73dbe45fabc3a3574f465bee3805bd773442b2d467822bb826f76da2e: Status 404 returned error can't find the container with id 416cc9f73dbe45fabc3a3574f465bee3805bd773442b2d467822bb826f76da2e Oct 29 22:15:59.977954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:15:59.977937 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:16:00.377129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.377094 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" event={"ID":"23715cbc-f9cd-496d-9e45-c2306e7aa75c","Type":"ContainerStarted","Data":"416cc9f73dbe45fabc3a3574f465bee3805bd773442b2d467822bb826f76da2e"} Oct 29 22:16:00.378322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.378296 5367 generic.go:356] "Generic (PLEG): container finished" podID="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" containerID="47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a" exitCode=0 Oct 29 22:16:00.378456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.378366 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf4d96c-bcdwj" Oct 29 22:16:00.378456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.378385 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf4d96c-bcdwj" event={"ID":"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae","Type":"ContainerDied","Data":"47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a"} Oct 29 22:16:00.378456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.378443 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf4d96c-bcdwj" event={"ID":"3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae","Type":"ContainerDied","Data":"873256a1837b3b39387b482fb8884aacaff0ced9ad2f598f4d13236c0b9d693e"} Oct 29 22:16:00.378559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.378469 5367 scope.go:122] "RemoveContainer" containerID="47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a" Oct 29 22:16:00.386933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.386812 5367 scope.go:122] "RemoveContainer" containerID="47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a" Oct 29 22:16:00.387226 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:16:00.387199 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a\": container with ID starting with 47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a not found: ID does not exist" containerID="47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a" Oct 29 22:16:00.387294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.387238 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a"} err="failed to get container status \"47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a\": rpc error: code = NotFound desc = could not find container \"47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a\": container with ID starting with 47fd5663cf77f05649fde6ea7d1660ad3f466ef336b9cdbf89a0fa3f8de7158a not found: ID does not exist" Oct 29 22:16:00.431149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.431126 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf4d96c-bcdwj"] Oct 29 22:16:00.434838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:00.434818 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bf4d96c-bcdwj"] Oct 29 22:16:01.118434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:01.118387 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7l9b5" Oct 29 22:16:01.219019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:01.218989 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" path="/var/lib/kubelet/pods/3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae/volumes" Oct 29 22:16:02.388710 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.388664 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" event={"ID":"23715cbc-f9cd-496d-9e45-c2306e7aa75c","Type":"ContainerStarted","Data":"e8ecc8d343690680e147fef06fe3f0698181afdfc98a04471a5bca9318f93376"} Oct 29 22:16:02.389122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.388737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" event={"ID":"23715cbc-f9cd-496d-9e45-c2306e7aa75c","Type":"ContainerStarted","Data":"04514b63397410612e5f92079d5b0391b9243ef53760a9bcd4330c24f236050c"} Oct 29 22:16:02.389122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.388758 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" event={"ID":"23715cbc-f9cd-496d-9e45-c2306e7aa75c","Type":"ContainerStarted","Data":"56b4b64313d4c21e1b684ed3be3fcd484772bf873bbbfa09491cd174d0ee8893"} Oct 29 22:16:02.461690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.461639 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-767dbcc698-ncbkm" podStartSLOduration=1.615237464 podStartE2EDuration="3.461624502s" totalImagesPullingTime="1.846387038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:02.461463191 +0000 UTC m=+29.355538111" watchObservedRunningTime="2026-10-29 22:16:02.461624502 +0000 UTC m=+29.355699423" Oct 29 22:16:02.613184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.613148 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59dcb976cb-zkxzg"] Oct 29 22:16:02.613415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.613403 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" containerName="console" Oct 29 22:16:02.613460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.613417 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" containerName="console" Oct 29 22:16:02.613493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.613478 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0c6f8b-6b6f-477d-b399-59c9bb3cb1ae" containerName="console" Oct 29 22:16:02.615306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.615290 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.637672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.637644 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59dcb976cb-zkxzg"] Oct 29 22:16:02.651322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651260 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651304 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651323 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651455 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651485 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgk8\" (UniqueName: \"kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651519 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651536 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.651734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.651640 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752359 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752398 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752428 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752452 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgk8\" (UniqueName: \"kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752481 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752497 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752516 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.752658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.752551 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.753081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.753048 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.753332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.753309 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.753332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.753319 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.753454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.753433 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.753511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.753497 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.755078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.755052 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.755078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.755072 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.773007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.772983 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgk8\" (UniqueName: \"kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8\") pod \"console-59dcb976cb-zkxzg\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:02.924949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:02.924862 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:03.218946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:03.218920 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59dcb976cb-zkxzg"] Oct 29 22:16:03.219058 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:16:03.218945 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e1689b_cd11_4cc9_8079_eff4297d8ddd.slice/crio-cf10e862bd3e66129ac51d621a808f3b582bf6f64334f30768c479f9e6bc9307 WatchSource:0}: Error finding container cf10e862bd3e66129ac51d621a808f3b582bf6f64334f30768c479f9e6bc9307: Status 404 returned error can't find the container with id cf10e862bd3e66129ac51d621a808f3b582bf6f64334f30768c479f9e6bc9307 Oct 29 22:16:03.393672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:03.393638 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcb976cb-zkxzg" event={"ID":"11e1689b-cd11-4cc9-8079-eff4297d8ddd","Type":"ContainerStarted","Data":"4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680"} Oct 29 22:16:03.394183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:03.393691 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcb976cb-zkxzg" event={"ID":"11e1689b-cd11-4cc9-8079-eff4297d8ddd","Type":"ContainerStarted","Data":"cf10e862bd3e66129ac51d621a808f3b582bf6f64334f30768c479f9e6bc9307"} Oct 29 22:16:03.436621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:03.436569 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59dcb976cb-zkxzg" podStartSLOduration=1.43655535 podStartE2EDuration="1.43655535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:03.435663311 +0000 UTC m=+30.329738232" watchObservedRunningTime="2026-10-29 22:16:03.43655535 +0000 UTC m=+30.330630271" Oct 29 22:16:12.925542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:12.925503 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:12.926105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:12.925598 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:12.930323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:12.930300 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:13.433174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:13.433146 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:16:13.505186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:13.505151 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b8fc9b66-29q5f"] Oct 29 22:16:22.329079 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:16:22.329040 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:16:38.529562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.529503 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67b8fc9b66-29q5f" podUID="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" containerName="console" containerID="cri-o://cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073" gracePeriod=15 Oct 29 22:16:38.760706 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.760669 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:16:38.836442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836407 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836450 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836479 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836506 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836542 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836578 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836617 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.836969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836653 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") pod \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\" (UID: \"34ac81cd-cb0f-49aa-b0d9-067dff8442b0\") " Oct 29 22:16:38.837019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.836982 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:38.837071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.837044 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:38.837127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.837082 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:38.837127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.837097 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:38.837219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.837153 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:38.839074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.839029 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:38.839167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.839072 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "kube-api-access-g98c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:38.839167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.839091 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config" pod "34ac81cd-cb0f-49aa-b0d9-067dff8442b0" (UID: "34ac81cd-cb0f-49aa-b0d9-067dff8442b0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:38.938254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938220 5367 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938250 5367 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-trusted-ca-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938260 5367 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938269 5367 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-oauth-serving-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938278 5367 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-serving-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938287 5367 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-tmp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938296 5367 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-console-oauth-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:38.938476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:38.938305 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98c2\" (UniqueName: \"kubernetes.io/projected/34ac81cd-cb0f-49aa-b0d9-067dff8442b0-kube-api-access-g98c2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:16:39.512512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.512436 5367 generic.go:356] "Generic (PLEG): container finished" podID="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" containerID="cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073" exitCode=0 Oct 29 22:16:39.512512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.512477 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b8fc9b66-29q5f" event={"ID":"34ac81cd-cb0f-49aa-b0d9-067dff8442b0","Type":"ContainerDied","Data":"cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073"} Oct 29 22:16:39.512742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.512518 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b8fc9b66-29q5f" event={"ID":"34ac81cd-cb0f-49aa-b0d9-067dff8442b0","Type":"ContainerDied","Data":"f30987dc08cddbefaedfdeeaf832c92cdfe1b19ab420fc72ae60ce79fdc7f51d"} Oct 29 22:16:39.512742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.512540 5367 scope.go:122] "RemoveContainer" containerID="cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073" Oct 29 22:16:39.512742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.512480 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b8fc9b66-29q5f" Oct 29 22:16:39.520118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.520089 5367 scope.go:122] "RemoveContainer" containerID="cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073" Oct 29 22:16:39.520405 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:16:39.520383 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073\": container with ID starting with cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073 not found: ID does not exist" containerID="cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073" Oct 29 22:16:39.520443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.520416 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073"} err="failed to get container status \"cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073\": rpc error: code = NotFound desc = could not find container \"cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073\": container with ID starting with cdeef028034c239862ffa1a6f4c99b664f823b759cb7faf36e85468e671d2073 not found: ID does not exist" Oct 29 22:16:39.567249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.567220 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b8fc9b66-29q5f"] Oct 29 22:16:39.580248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:39.580217 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67b8fc9b66-29q5f"] Oct 29 22:16:41.218340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:16:41.218309 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" path="/var/lib/kubelet/pods/34ac81cd-cb0f-49aa-b0d9-067dff8442b0/volumes" Oct 29 22:16:52.329273 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:16:52.329237 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:17:20.674367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.674286 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-55dffb84cd-26pgk"] Oct 29 22:17:20.674758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.674553 5367 cpu_manager.go:414] "containerMap: removing container" podUID="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" containerName="console" Oct 29 22:17:20.674758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.674563 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" containerName="console" Oct 29 22:17:20.674758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.674603 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ac81cd-cb0f-49aa-b0d9-067dff8442b0" containerName="console" Oct 29 22:17:20.676397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.676381 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.709229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.709205 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55dffb84cd-26pgk"] Oct 29 22:17:20.749066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749043 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-service-ca\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749072 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-oauth-serving-cert\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749095 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-serving-cert\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749168 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-tmp\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749266 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7mf\" (UniqueName: \"kubernetes.io/projected/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-kube-api-access-8v7mf\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749290 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-config\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749315 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-oauth-config\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.749386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.749330 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-trusted-ca-bundle\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850342 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7mf\" (UniqueName: \"kubernetes.io/projected/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-kube-api-access-8v7mf\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850379 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-config\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850406 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-oauth-config\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850428 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-trusted-ca-bundle\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850471 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-service-ca\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850493 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-oauth-serving-cert\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850528 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-serving-cert\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.850807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.850574 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-tmp\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.851046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.851019 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-tmp\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.851294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.851271 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-service-ca\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.851384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.851334 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-oauth-serving-cert\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.851447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.851412 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-trusted-ca-bundle\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.851447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.851436 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-config\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.852996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.852966 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-oauth-config\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.853184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.853163 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-console-serving-cert\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.871729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.871695 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7mf\" (UniqueName: \"kubernetes.io/projected/fdb0aae7-06f9-4d61-8f4a-a46ce846917b-kube-api-access-8v7mf\") pod \"console-55dffb84cd-26pgk\" (UID: \"fdb0aae7-06f9-4d61-8f4a-a46ce846917b\") " pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:20.986558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:20.986485 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:21.278672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:21.278598 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55dffb84cd-26pgk"] Oct 29 22:17:21.281881 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:17:21.281855 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb0aae7_06f9_4d61_8f4a_a46ce846917b.slice/crio-98bc58cc915124b95edaf24d3acd23a30bb1044e1ae5fe2b00f3b1d65eff342f WatchSource:0}: Error finding container 98bc58cc915124b95edaf24d3acd23a30bb1044e1ae5fe2b00f3b1d65eff342f: Status 404 returned error can't find the container with id 98bc58cc915124b95edaf24d3acd23a30bb1044e1ae5fe2b00f3b1d65eff342f Oct 29 22:17:21.646063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:21.646022 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dffb84cd-26pgk" event={"ID":"fdb0aae7-06f9-4d61-8f4a-a46ce846917b","Type":"ContainerStarted","Data":"d3c62eba0277642f4c2cd13fec4237df56c412b20b01962b913fe4d3c56d83a8"} Oct 29 22:17:21.646229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:21.646077 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dffb84cd-26pgk" event={"ID":"fdb0aae7-06f9-4d61-8f4a-a46ce846917b","Type":"ContainerStarted","Data":"98bc58cc915124b95edaf24d3acd23a30bb1044e1ae5fe2b00f3b1d65eff342f"} Oct 29 22:17:21.689545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:21.689405 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-55dffb84cd-26pgk" podStartSLOduration=1.689389046 podStartE2EDuration="1.689389046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:21.688507599 +0000 UTC m=+108.582582519" watchObservedRunningTime="2026-10-29 22:17:21.689389046 +0000 UTC m=+108.583463967" Oct 29 22:17:22.328236 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:17:22.328197 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:17:30.986614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:30.986578 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:30.987122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:30.986647 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:30.991386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:30.991360 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:31.682062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:31.682034 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55dffb84cd-26pgk" Oct 29 22:17:31.749983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:31.749951 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59dcb976cb-zkxzg"] Oct 29 22:17:52.329275 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:17:52.329236 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:17:56.772436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:56.772395 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59dcb976cb-zkxzg" podUID="11e1689b-cd11-4cc9-8079-eff4297d8ddd" containerName="console" containerID="cri-o://4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680" gracePeriod=15 Oct 29 22:17:57.008470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.008443 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:17:57.126563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126532 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126569 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126592 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126613 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126780 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8\" (UniqueName: \"kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126828 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126871 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.126929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126898 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config\") pod \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\" (UID: \"11e1689b-cd11-4cc9-8079-eff4297d8ddd\") " Oct 29 22:17:57.127130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.126918 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:57.127130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127013 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:57.127130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127053 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:57.127299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127282 5367 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.127359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127306 5367 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-trusted-ca-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.127359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127319 5367 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11e1689b-cd11-4cc9-8079-eff4297d8ddd-tmp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.127359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127333 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:57.127359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.127339 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:57.128915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.128894 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "kube-api-access-4xgk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:57.129333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.129306 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:17:57.129372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.129317 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config" pod "11e1689b-cd11-4cc9-8079-eff4297d8ddd" (UID: "11e1689b-cd11-4cc9-8079-eff4297d8ddd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:17:57.228035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.228017 5367 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-oauth-serving-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.228035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.228036 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xgk8\" (UniqueName: \"kubernetes.io/projected/11e1689b-cd11-4cc9-8079-eff4297d8ddd-kube-api-access-4xgk8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.228171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.228045 5367 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-serving-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.228171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.228053 5367 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-oauth-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.228171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.228062 5367 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11e1689b-cd11-4cc9-8079-eff4297d8ddd-console-config\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:17:57.760441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.760405 5367 generic.go:356] "Generic (PLEG): container finished" podID="11e1689b-cd11-4cc9-8079-eff4297d8ddd" containerID="4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680" exitCode=0 Oct 29 22:17:57.760614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.760458 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcb976cb-zkxzg" Oct 29 22:17:57.760614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.760481 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcb976cb-zkxzg" event={"ID":"11e1689b-cd11-4cc9-8079-eff4297d8ddd","Type":"ContainerDied","Data":"4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680"} Oct 29 22:17:57.760614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.760521 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcb976cb-zkxzg" event={"ID":"11e1689b-cd11-4cc9-8079-eff4297d8ddd","Type":"ContainerDied","Data":"cf10e862bd3e66129ac51d621a808f3b582bf6f64334f30768c479f9e6bc9307"} Oct 29 22:17:57.760614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.760540 5367 scope.go:122] "RemoveContainer" containerID="4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680" Oct 29 22:17:57.767996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.767977 5367 scope.go:122] "RemoveContainer" containerID="4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680" Oct 29 22:17:57.768272 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:17:57.768252 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680\": container with ID starting with 4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680 not found: ID does not exist" containerID="4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680" Oct 29 22:17:57.768342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.768282 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680"} err="failed to get container status \"4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680\": rpc error: code = NotFound desc = could not find container \"4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680\": container with ID starting with 4902de96d98aeb36d9d22b49617f54e8d0e33171af965119be35d4bbb77b1680 not found: ID does not exist" Oct 29 22:17:57.815588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.815553 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59dcb976cb-zkxzg"] Oct 29 22:17:57.829647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:57.829620 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59dcb976cb-zkxzg"] Oct 29 22:17:59.217898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:17:59.217867 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e1689b-cd11-4cc9-8079-eff4297d8ddd" path="/var/lib/kubelet/pods/11e1689b-cd11-4cc9-8079-eff4297d8ddd/volumes" Oct 29 22:18:22.328998 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:18:22.328956 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:18:52.329646 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:18:52.329571 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:19:20.568571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.568537 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp"] Oct 29 22:19:20.569031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.568923 5367 cpu_manager.go:414] "containerMap: removing container" podUID="11e1689b-cd11-4cc9-8079-eff4297d8ddd" containerName="console" Oct 29 22:19:20.569031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.568943 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1689b-cd11-4cc9-8079-eff4297d8ddd" containerName="console" Oct 29 22:19:20.569031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.569006 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1689b-cd11-4cc9-8079-eff4297d8ddd" containerName="console" Oct 29 22:19:20.572017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.571997 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.588811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.588789 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 22:19:20.602790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.602767 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 22:19:20.602893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.602799 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-jtpfd\"" Oct 29 22:19:20.603479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.603462 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp"] Oct 29 22:19:20.721236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.721203 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.721236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.721240 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7bp\" (UniqueName: \"kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.721429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.721272 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.822027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.821969 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.822027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.822009 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7bp\" (UniqueName: \"kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.822128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.822039 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.822362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.822338 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.822401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.822369 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.843169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.843141 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7bp\" (UniqueName: \"kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:20.887440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:20.887420 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:21.179265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:21.179185 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp"] Oct 29 22:19:21.182564 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:19:21.182525 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8ec821_f42a_4595_8c1c_e15bece3ace8.slice/crio-a57d2d1f1523090307ffef3e3e4707893f088253744c117760815db724bbd705 WatchSource:0}: Error finding container a57d2d1f1523090307ffef3e3e4707893f088253744c117760815db724bbd705: Status 404 returned error can't find the container with id a57d2d1f1523090307ffef3e3e4707893f088253744c117760815db724bbd705 Oct 29 22:19:22.037275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:22.037119 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" event={"ID":"8e8ec821-f42a-4595-8c1c-e15bece3ace8","Type":"ContainerStarted","Data":"a57d2d1f1523090307ffef3e3e4707893f088253744c117760815db724bbd705"} Oct 29 22:19:22.335269 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:19:22.335228 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:19:27.053960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:27.053924 5367 generic.go:356] "Generic (PLEG): container finished" podID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerID="8b6e7ea981a0909714dd77d94abcc4dbd3b8091daa12102bfaae048463ca0b2e" exitCode=0 Oct 29 22:19:27.054382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:27.054021 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" event={"ID":"8e8ec821-f42a-4595-8c1c-e15bece3ace8","Type":"ContainerDied","Data":"8b6e7ea981a0909714dd77d94abcc4dbd3b8091daa12102bfaae048463ca0b2e"} Oct 29 22:19:29.061598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:29.061567 5367 generic.go:356] "Generic (PLEG): container finished" podID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerID="282b9ebee2da38c0a6de882bb6f0916fdee7eed3c6973b9f54c81835110f9906" exitCode=0 Oct 29 22:19:29.061975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:29.061639 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" event={"ID":"8e8ec821-f42a-4595-8c1c-e15bece3ace8","Type":"ContainerDied","Data":"282b9ebee2da38c0a6de882bb6f0916fdee7eed3c6973b9f54c81835110f9906"} Oct 29 22:19:35.086372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:35.086338 5367 generic.go:356] "Generic (PLEG): container finished" podID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerID="cafd02c58089a1a31760edb2fedf4e577fabdfeec13a36404306d3f33e627316" exitCode=0 Oct 29 22:19:35.086766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:35.086379 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" event={"ID":"8e8ec821-f42a-4595-8c1c-e15bece3ace8","Type":"ContainerDied","Data":"cafd02c58089a1a31760edb2fedf4e577fabdfeec13a36404306d3f33e627316"} Oct 29 22:19:36.208660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.208636 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:36.249257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.249217 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp\" (UniqueName: \"kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp\") pod \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " Oct 29 22:19:36.249439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.249266 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle\") pod \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " Oct 29 22:19:36.249439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.249306 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util\") pod \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\" (UID: \"8e8ec821-f42a-4595-8c1c-e15bece3ace8\") " Oct 29 22:19:36.249921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.249889 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle" pod "8e8ec821-f42a-4595-8c1c-e15bece3ace8" (UID: "8e8ec821-f42a-4595-8c1c-e15bece3ace8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:36.251680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.251625 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp" pod "8e8ec821-f42a-4595-8c1c-e15bece3ace8" (UID: "8e8ec821-f42a-4595-8c1c-e15bece3ace8"). InnerVolumeSpecName "kube-api-access-rt7bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:36.254622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.254594 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util" pod "8e8ec821-f42a-4595-8c1c-e15bece3ace8" (UID: "8e8ec821-f42a-4595-8c1c-e15bece3ace8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:36.350858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.350777 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7bp\" (UniqueName: \"kubernetes.io/projected/8e8ec821-f42a-4595-8c1c-e15bece3ace8-kube-api-access-rt7bp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:19:36.350858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.350810 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:19:36.350858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:36.350821 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e8ec821-f42a-4595-8c1c-e15bece3ace8-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:19:37.094512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:37.094471 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" event={"ID":"8e8ec821-f42a-4595-8c1c-e15bece3ace8","Type":"ContainerDied","Data":"a57d2d1f1523090307ffef3e3e4707893f088253744c117760815db724bbd705"} Oct 29 22:19:37.094682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:37.094529 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2mdpp" Oct 29 22:19:37.094682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:37.094560 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57d2d1f1523090307ffef3e3e4707893f088253744c117760815db724bbd705" Oct 29 22:19:43.449437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449400 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg"] Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449656 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="extract" Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449667 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="extract" Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449689 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="util" Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449694 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="util" Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449726 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="pull" Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449733 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="pull" Oct 29 22:19:43.449865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.449779 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8ec821-f42a-4595-8c1c-e15bece3ace8" containerName="extract" Oct 29 22:19:43.451961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.451945 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.471797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.471766 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 29 22:19:43.471995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.471766 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:19:43.485854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.485807 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7pnzz\"" Oct 29 22:19:43.486650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.486611 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg"] Oct 29 22:19:43.505568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.505528 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95118ffe-b591-483e-8a6f-2cba3952ac27-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-d2wtg\" (UID: \"95118ffe-b591-483e-8a6f-2cba3952ac27\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.505782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.505600 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzbg\" (UniqueName: \"kubernetes.io/projected/95118ffe-b591-483e-8a6f-2cba3952ac27-kube-api-access-5fzbg\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-d2wtg\" (UID: \"95118ffe-b591-483e-8a6f-2cba3952ac27\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.606454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.606412 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95118ffe-b591-483e-8a6f-2cba3952ac27-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-d2wtg\" (UID: \"95118ffe-b591-483e-8a6f-2cba3952ac27\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.606631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.606472 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzbg\" (UniqueName: \"kubernetes.io/projected/95118ffe-b591-483e-8a6f-2cba3952ac27-kube-api-access-5fzbg\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-d2wtg\" (UID: \"95118ffe-b591-483e-8a6f-2cba3952ac27\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.606851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.606829 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95118ffe-b591-483e-8a6f-2cba3952ac27-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-d2wtg\" (UID: \"95118ffe-b591-483e-8a6f-2cba3952ac27\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.627577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.627538 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzbg\" (UniqueName: \"kubernetes.io/projected/95118ffe-b591-483e-8a6f-2cba3952ac27-kube-api-access-5fzbg\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-d2wtg\" (UID: \"95118ffe-b591-483e-8a6f-2cba3952ac27\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:43.762679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:43.762603 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" Oct 29 22:19:44.088459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:44.088423 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg"] Oct 29 22:19:44.104653 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:19:44.104619 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95118ffe_b591_483e_8a6f_2cba3952ac27.slice/crio-ebb78ccbfe353ab69db0116a2c990d2a879c0d3a2b305a270772fdc5dc6f7582 WatchSource:0}: Error finding container ebb78ccbfe353ab69db0116a2c990d2a879c0d3a2b305a270772fdc5dc6f7582: Status 404 returned error can't find the container with id ebb78ccbfe353ab69db0116a2c990d2a879c0d3a2b305a270772fdc5dc6f7582 Oct 29 22:19:44.118183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:44.118150 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" event={"ID":"95118ffe-b591-483e-8a6f-2cba3952ac27","Type":"ContainerStarted","Data":"ebb78ccbfe353ab69db0116a2c990d2a879c0d3a2b305a270772fdc5dc6f7582"} Oct 29 22:19:46.127767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:46.127726 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" event={"ID":"95118ffe-b591-483e-8a6f-2cba3952ac27","Type":"ContainerStarted","Data":"7b5ddfa751cdb68431906ec50e6868d578a6b9ed63fd2e1400463868e3a2f96a"} Oct 29 22:19:46.200863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:46.200813 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-d2wtg" podStartSLOduration=1.6793932219999999 podStartE2EDuration="3.200799972s" totalImagesPullingTime="1.52140675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:46.186848095 +0000 UTC m=+253.080923018" watchObservedRunningTime="2026-10-29 22:19:46.200799972 +0000 UTC m=+253.094874893" Oct 29 22:19:47.167766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.167730 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf"] Oct 29 22:19:47.170215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.170192 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.200479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.200437 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-r5vfl\"" Oct 29 22:19:47.200647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.200532 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 22:19:47.200647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.200621 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 22:19:47.201617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.201597 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf"] Oct 29 22:19:47.233390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.233358 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8q6b\" (UniqueName: \"kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.233500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.233439 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.233563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.233508 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.251519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.251496 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp"] Oct 29 22:19:47.253547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.253529 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.269859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.269844 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jpw2j\"" Oct 29 22:19:47.275667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.275642 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp"] Oct 29 22:19:47.286419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.286404 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 29 22:19:47.286510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.286476 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 29 22:19:47.334196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334169 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6ns\" (UniqueName: \"kubernetes.io/projected/fd4be442-0fd5-483a-a2fb-45561dfdaafc-kube-api-access-tm6ns\") pod \"cert-manager-webhook-5dbbbf75cc-krfjp\" (UID: \"fd4be442-0fd5-483a-a2fb-45561dfdaafc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.334285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334236 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q6b\" (UniqueName: \"kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.334362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334282 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd4be442-0fd5-483a-a2fb-45561dfdaafc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krfjp\" (UID: \"fd4be442-0fd5-483a-a2fb-45561dfdaafc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.334362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334323 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.334362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334356 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.334652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334636 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.334694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.334672 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.355396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.355372 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8q6b\" (UniqueName: \"kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.435475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.435408 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd4be442-0fd5-483a-a2fb-45561dfdaafc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krfjp\" (UID: \"fd4be442-0fd5-483a-a2fb-45561dfdaafc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.435475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.435461 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6ns\" (UniqueName: \"kubernetes.io/projected/fd4be442-0fd5-483a-a2fb-45561dfdaafc-kube-api-access-tm6ns\") pod \"cert-manager-webhook-5dbbbf75cc-krfjp\" (UID: \"fd4be442-0fd5-483a-a2fb-45561dfdaafc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.457051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.457016 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd4be442-0fd5-483a-a2fb-45561dfdaafc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krfjp\" (UID: \"fd4be442-0fd5-483a-a2fb-45561dfdaafc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.457244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.457228 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6ns\" (UniqueName: \"kubernetes.io/projected/fd4be442-0fd5-483a-a2fb-45561dfdaafc-kube-api-access-tm6ns\") pod \"cert-manager-webhook-5dbbbf75cc-krfjp\" (UID: \"fd4be442-0fd5-483a-a2fb-45561dfdaafc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.482585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.482565 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:47.572025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.571995 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:47.786711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.786616 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf"] Oct 29 22:19:47.790077 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:19:47.790041 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5345c060_e156_40eb_9ff2_bd6edc35478c.slice/crio-1425a57ba389b3a2b3957bf84c7072d94b6e7ec9f2e6b156d30753f90d88879d WatchSource:0}: Error finding container 1425a57ba389b3a2b3957bf84c7072d94b6e7ec9f2e6b156d30753f90d88879d: Status 404 returned error can't find the container with id 1425a57ba389b3a2b3957bf84c7072d94b6e7ec9f2e6b156d30753f90d88879d Oct 29 22:19:47.861640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:47.861610 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp"] Oct 29 22:19:47.865478 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:19:47.865445 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd4be442_0fd5_483a_a2fb_45561dfdaafc.slice/crio-16670f4bd6f5a5f2bd20d7fd62bb2246f6ba50af554131e5e40d582faee96566 WatchSource:0}: Error finding container 16670f4bd6f5a5f2bd20d7fd62bb2246f6ba50af554131e5e40d582faee96566: Status 404 returned error can't find the container with id 16670f4bd6f5a5f2bd20d7fd62bb2246f6ba50af554131e5e40d582faee96566 Oct 29 22:19:48.136465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:48.136423 5367 generic.go:356] "Generic (PLEG): container finished" podID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerID="c8284c4ac5a60fc24870c1f7c4fcfbdea0f91dcb27a3eed1eb78a4cc584abc14" exitCode=0 Oct 29 22:19:48.136659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:48.136506 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" event={"ID":"5345c060-e156-40eb-9ff2-bd6edc35478c","Type":"ContainerDied","Data":"c8284c4ac5a60fc24870c1f7c4fcfbdea0f91dcb27a3eed1eb78a4cc584abc14"} Oct 29 22:19:48.136659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:48.136565 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" event={"ID":"5345c060-e156-40eb-9ff2-bd6edc35478c","Type":"ContainerStarted","Data":"1425a57ba389b3a2b3957bf84c7072d94b6e7ec9f2e6b156d30753f90d88879d"} Oct 29 22:19:48.138742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:48.138717 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" event={"ID":"fd4be442-0fd5-483a-a2fb-45561dfdaafc","Type":"ContainerStarted","Data":"16670f4bd6f5a5f2bd20d7fd62bb2246f6ba50af554131e5e40d582faee96566"} Oct 29 22:19:51.154921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:51.154887 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" event={"ID":"fd4be442-0fd5-483a-a2fb-45561dfdaafc","Type":"ContainerStarted","Data":"4129c186f12fdb02f03079fd3fefecae910de99f48a71f0b9af530be08b9b315"} Oct 29 22:19:51.155378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:51.154985 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:51.156686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:51.156663 5367 generic.go:356] "Generic (PLEG): container finished" podID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerID="ff7d9653e7b0752384e57698d787ba38f32805a80f71852e6cecf521268c68ee" exitCode=0 Oct 29 22:19:51.156795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:51.156731 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" event={"ID":"5345c060-e156-40eb-9ff2-bd6edc35478c","Type":"ContainerDied","Data":"ff7d9653e7b0752384e57698d787ba38f32805a80f71852e6cecf521268c68ee"} Oct 29 22:19:51.196539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:51.196445 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" podStartSLOduration=1.650093579 podStartE2EDuration="4.196427767s" totalImagesPullingTime="2.546334188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:51.195585008 +0000 UTC m=+258.089659928" watchObservedRunningTime="2026-10-29 22:19:51.196427767 +0000 UTC m=+258.090502690" Oct 29 22:19:52.114856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.114822 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk"] Oct 29 22:19:52.116854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.116838 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.133303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.133282 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jh7f4\"" Oct 29 22:19:52.147823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.147794 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk"] Oct 29 22:19:52.162064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.162038 5367 generic.go:356] "Generic (PLEG): container finished" podID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerID="d1c6c315fab51d87e509c280dcf59f51b9388fc1f8d1669af0103044e9f72197" exitCode=0 Oct 29 22:19:52.162383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.162102 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" event={"ID":"5345c060-e156-40eb-9ff2-bd6edc35478c","Type":"ContainerDied","Data":"d1c6c315fab51d87e509c280dcf59f51b9388fc1f8d1669af0103044e9f72197"} Oct 29 22:19:52.175881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.175845 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f03d45ca-365c-4eb3-8401-64178d11d144-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vkcjk\" (UID: \"f03d45ca-365c-4eb3-8401-64178d11d144\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.175978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.175946 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdjx\" (UniqueName: \"kubernetes.io/projected/f03d45ca-365c-4eb3-8401-64178d11d144-kube-api-access-kzdjx\") pod \"cert-manager-cainjector-5c57bbd4fc-vkcjk\" (UID: \"f03d45ca-365c-4eb3-8401-64178d11d144\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.276772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.276743 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f03d45ca-365c-4eb3-8401-64178d11d144-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vkcjk\" (UID: \"f03d45ca-365c-4eb3-8401-64178d11d144\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.276882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.276792 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdjx\" (UniqueName: \"kubernetes.io/projected/f03d45ca-365c-4eb3-8401-64178d11d144-kube-api-access-kzdjx\") pod \"cert-manager-cainjector-5c57bbd4fc-vkcjk\" (UID: \"f03d45ca-365c-4eb3-8401-64178d11d144\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.298133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.298108 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdjx\" (UniqueName: \"kubernetes.io/projected/f03d45ca-365c-4eb3-8401-64178d11d144-kube-api-access-kzdjx\") pod \"cert-manager-cainjector-5c57bbd4fc-vkcjk\" (UID: \"f03d45ca-365c-4eb3-8401-64178d11d144\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.298133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.298123 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f03d45ca-365c-4eb3-8401-64178d11d144-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vkcjk\" (UID: \"f03d45ca-365c-4eb3-8401-64178d11d144\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.328297 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:19:52.328268 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:19:52.426832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.426773 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" Oct 29 22:19:52.714882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:52.714795 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk"] Oct 29 22:19:52.718341 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:19:52.718315 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf03d45ca_365c_4eb3_8401_64178d11d144.slice/crio-7351677d3fafa46212a3cb2957d07893205a7ff49bd9db6f557535faa7f3b919 WatchSource:0}: Error finding container 7351677d3fafa46212a3cb2957d07893205a7ff49bd9db6f557535faa7f3b919: Status 404 returned error can't find the container with id 7351677d3fafa46212a3cb2957d07893205a7ff49bd9db6f557535faa7f3b919 Oct 29 22:19:53.167652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.167617 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" event={"ID":"f03d45ca-365c-4eb3-8401-64178d11d144","Type":"ContainerStarted","Data":"7d87101509605ad5ca4cf80a68b5f8dc4011132486094106fdbdcc323b71e021"} Oct 29 22:19:53.168083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.167664 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" event={"ID":"f03d45ca-365c-4eb3-8401-64178d11d144","Type":"ContainerStarted","Data":"7351677d3fafa46212a3cb2957d07893205a7ff49bd9db6f557535faa7f3b919"} Oct 29 22:19:53.209669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.209616 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vkcjk" podStartSLOduration=1.209600599 podStartE2EDuration="1.209600599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:53.208332127 +0000 UTC m=+260.102407049" watchObservedRunningTime="2026-10-29 22:19:53.209600599 +0000 UTC m=+260.103675520" Oct 29 22:19:53.302354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.302330 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:53.388086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.388058 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util\") pod \"5345c060-e156-40eb-9ff2-bd6edc35478c\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " Oct 29 22:19:53.388215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.388111 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle\") pod \"5345c060-e156-40eb-9ff2-bd6edc35478c\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " Oct 29 22:19:53.388215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.388145 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b\" (UniqueName: \"kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b\") pod \"5345c060-e156-40eb-9ff2-bd6edc35478c\" (UID: \"5345c060-e156-40eb-9ff2-bd6edc35478c\") " Oct 29 22:19:53.388485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.388464 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle" pod "5345c060-e156-40eb-9ff2-bd6edc35478c" (UID: "5345c060-e156-40eb-9ff2-bd6edc35478c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:53.390368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.390343 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b" pod "5345c060-e156-40eb-9ff2-bd6edc35478c" (UID: "5345c060-e156-40eb-9ff2-bd6edc35478c"). InnerVolumeSpecName "kube-api-access-j8q6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:53.392285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.392260 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util" pod "5345c060-e156-40eb-9ff2-bd6edc35478c" (UID: "5345c060-e156-40eb-9ff2-bd6edc35478c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:53.488969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.488943 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:19:53.488969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.488966 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5345c060-e156-40eb-9ff2-bd6edc35478c-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:19:53.489085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:53.488975 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8q6b\" (UniqueName: \"kubernetes.io/projected/5345c060-e156-40eb-9ff2-bd6edc35478c-kube-api-access-j8q6b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.173289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:54.173258 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" Oct 29 22:19:54.173289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:54.173281 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85bz6hf" event={"ID":"5345c060-e156-40eb-9ff2-bd6edc35478c","Type":"ContainerDied","Data":"1425a57ba389b3a2b3957bf84c7072d94b6e7ec9f2e6b156d30753f90d88879d"} Oct 29 22:19:54.173665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:54.173326 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1425a57ba389b3a2b3957bf84c7072d94b6e7ec9f2e6b156d30753f90d88879d" Oct 29 22:19:57.164440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:57.164410 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" Oct 29 22:19:58.793403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793365 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-9pb8d"] Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793797 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="extract" Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793816 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="extract" Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793854 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="pull" Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793862 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="pull" Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793887 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="util" Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793895 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="util" Oct 29 22:19:58.794130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.793960 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5345c060-e156-40eb-9ff2-bd6edc35478c" containerName="extract" Oct 29 22:19:58.798857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.798837 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:58.817045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.817019 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9pb8d"] Oct 29 22:19:58.829568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.829551 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9r4c5\"" Oct 29 22:19:58.930984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.930958 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5jc\" (UniqueName: \"kubernetes.io/projected/c905be47-b83d-4cee-8b3c-c1a89acb9f9b-kube-api-access-5s5jc\") pod \"cert-manager-597b95cb84-9pb8d\" (UID: \"c905be47-b83d-4cee-8b3c-c1a89acb9f9b\") " pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:58.931108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:58.931027 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c905be47-b83d-4cee-8b3c-c1a89acb9f9b-bound-sa-token\") pod \"cert-manager-597b95cb84-9pb8d\" (UID: \"c905be47-b83d-4cee-8b3c-c1a89acb9f9b\") " pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:59.032327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:59.032296 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c905be47-b83d-4cee-8b3c-c1a89acb9f9b-bound-sa-token\") pod \"cert-manager-597b95cb84-9pb8d\" (UID: \"c905be47-b83d-4cee-8b3c-c1a89acb9f9b\") " pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:59.032327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:59.032329 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5jc\" (UniqueName: \"kubernetes.io/projected/c905be47-b83d-4cee-8b3c-c1a89acb9f9b-kube-api-access-5s5jc\") pod \"cert-manager-597b95cb84-9pb8d\" (UID: \"c905be47-b83d-4cee-8b3c-c1a89acb9f9b\") " pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:59.053320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:59.053255 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c905be47-b83d-4cee-8b3c-c1a89acb9f9b-bound-sa-token\") pod \"cert-manager-597b95cb84-9pb8d\" (UID: \"c905be47-b83d-4cee-8b3c-c1a89acb9f9b\") " pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:59.053502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:59.053483 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5jc\" (UniqueName: \"kubernetes.io/projected/c905be47-b83d-4cee-8b3c-c1a89acb9f9b-kube-api-access-5s5jc\") pod \"cert-manager-597b95cb84-9pb8d\" (UID: \"c905be47-b83d-4cee-8b3c-c1a89acb9f9b\") " pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:59.109464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:59.109433 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9pb8d" Oct 29 22:19:59.398660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:19:59.398628 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9pb8d"] Oct 29 22:19:59.402061 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:19:59.402037 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc905be47_b83d_4cee_8b3c_c1a89acb9f9b.slice/crio-5f30d3823c1ce0bfb7bd015e764d6c58dc91c3a43657760382bc1bd79d8ce8a4 WatchSource:0}: Error finding container 5f30d3823c1ce0bfb7bd015e764d6c58dc91c3a43657760382bc1bd79d8ce8a4: Status 404 returned error can't find the container with id 5f30d3823c1ce0bfb7bd015e764d6c58dc91c3a43657760382bc1bd79d8ce8a4 Oct 29 22:20:00.165755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.165722 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg"] Oct 29 22:20:00.168897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.168881 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.196751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.196719 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9pb8d" event={"ID":"c905be47-b83d-4cee-8b3c-c1a89acb9f9b","Type":"ContainerStarted","Data":"a13df8553f826f52d8e0cfd3a875729de792f4a955e7eb516b9ffcc5365d51c8"} Oct 29 22:20:00.196914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.196767 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9pb8d" event={"ID":"c905be47-b83d-4cee-8b3c-c1a89acb9f9b","Type":"ContainerStarted","Data":"5f30d3823c1ce0bfb7bd015e764d6c58dc91c3a43657760382bc1bd79d8ce8a4"} Oct 29 22:20:00.199153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.199129 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-n9g7w\"" Oct 29 22:20:00.199153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.199144 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 29 22:20:00.199308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.199171 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:20:00.200155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.200138 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg"] Oct 29 22:20:00.241781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.241756 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbpz\" (UniqueName: \"kubernetes.io/projected/da0c9d49-7180-4af9-8269-bdfb0164d001-kube-api-access-pcbpz\") pod \"openshift-lws-operator-8465994787-rjnkg\" (UID: \"da0c9d49-7180-4af9-8269-bdfb0164d001\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.241921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.241817 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da0c9d49-7180-4af9-8269-bdfb0164d001-tmp\") pod \"openshift-lws-operator-8465994787-rjnkg\" (UID: \"da0c9d49-7180-4af9-8269-bdfb0164d001\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.255087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.255041 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-9pb8d" podStartSLOduration=2.25502883 podStartE2EDuration="2.25502883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:00.254504466 +0000 UTC m=+267.148579386" watchObservedRunningTime="2026-10-29 22:20:00.25502883 +0000 UTC m=+267.149103751" Oct 29 22:20:00.342708 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.342672 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da0c9d49-7180-4af9-8269-bdfb0164d001-tmp\") pod \"openshift-lws-operator-8465994787-rjnkg\" (UID: \"da0c9d49-7180-4af9-8269-bdfb0164d001\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.342880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.342794 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbpz\" (UniqueName: \"kubernetes.io/projected/da0c9d49-7180-4af9-8269-bdfb0164d001-kube-api-access-pcbpz\") pod \"openshift-lws-operator-8465994787-rjnkg\" (UID: \"da0c9d49-7180-4af9-8269-bdfb0164d001\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.343136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.343114 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da0c9d49-7180-4af9-8269-bdfb0164d001-tmp\") pod \"openshift-lws-operator-8465994787-rjnkg\" (UID: \"da0c9d49-7180-4af9-8269-bdfb0164d001\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.363893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.363870 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbpz\" (UniqueName: \"kubernetes.io/projected/da0c9d49-7180-4af9-8269-bdfb0164d001-kube-api-access-pcbpz\") pod \"openshift-lws-operator-8465994787-rjnkg\" (UID: \"da0c9d49-7180-4af9-8269-bdfb0164d001\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.480327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.480241 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" Oct 29 22:20:00.775165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:00.775086 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg"] Oct 29 22:20:00.778234 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:20:00.778201 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda0c9d49_7180_4af9_8269_bdfb0164d001.slice/crio-fadc39aa0b061862e6033a0b9a902899523e5915c82755834ec5a5bbba6527f8 WatchSource:0}: Error finding container fadc39aa0b061862e6033a0b9a902899523e5915c82755834ec5a5bbba6527f8: Status 404 returned error can't find the container with id fadc39aa0b061862e6033a0b9a902899523e5915c82755834ec5a5bbba6527f8 Oct 29 22:20:01.201934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:01.201890 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" event={"ID":"da0c9d49-7180-4af9-8269-bdfb0164d001","Type":"ContainerStarted","Data":"fadc39aa0b061862e6033a0b9a902899523e5915c82755834ec5a5bbba6527f8"} Oct 29 22:20:03.212684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:03.212649 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" event={"ID":"da0c9d49-7180-4af9-8269-bdfb0164d001","Type":"ContainerStarted","Data":"2fc42a249400dfe94d3aac7b091ec4218987684243187bedd93d813db8e02e4f"} Oct 29 22:20:03.255211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:03.255112 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-rjnkg" podStartSLOduration=1.039129903 podStartE2EDuration="3.255097102s" totalImagesPullingTime="2.215967199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:03.253792232 +0000 UTC m=+270.147867153" watchObservedRunningTime="2026-10-29 22:20:03.255097102 +0000 UTC m=+270.149172024" Oct 29 22:20:06.621622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.621584 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl"] Oct 29 22:20:06.640855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.640831 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.645330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.645302 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl"] Oct 29 22:20:06.658497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.658478 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 22:20:06.658601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.658495 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 22:20:06.671919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.671899 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-z9v8w\"" Oct 29 22:20:06.695772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.695747 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.695930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.695785 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.695970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.695937 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nks4w\" (UniqueName: \"kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.796468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.796436 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nks4w\" (UniqueName: \"kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.796608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.796475 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.796608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.796503 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.796945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.796925 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.796992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.796936 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.817216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.817192 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nks4w\" (UniqueName: \"kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:06.991831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:06.991743 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:07.297467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:07.297323 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl"] Oct 29 22:20:07.299888 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:20:07.299859 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1dfa46_a410_41da_9a32_feca4887d12f.slice/crio-256e938578634d0f6c40a4207db81f730184fa30c0e8a67e33ab6936caefb3f3 WatchSource:0}: Error finding container 256e938578634d0f6c40a4207db81f730184fa30c0e8a67e33ab6936caefb3f3: Status 404 returned error can't find the container with id 256e938578634d0f6c40a4207db81f730184fa30c0e8a67e33ab6936caefb3f3 Oct 29 22:20:08.232908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:08.232805 5367 generic.go:356] "Generic (PLEG): container finished" podID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerID="8f930799700f1f99642ca8d3f701f63c9e57fa7df8bc7197969e0c1bb6f1a939" exitCode=0 Oct 29 22:20:08.232908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:08.232875 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" event={"ID":"0b1dfa46-a410-41da-9a32-feca4887d12f","Type":"ContainerDied","Data":"8f930799700f1f99642ca8d3f701f63c9e57fa7df8bc7197969e0c1bb6f1a939"} Oct 29 22:20:08.233420 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:08.232927 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" event={"ID":"0b1dfa46-a410-41da-9a32-feca4887d12f","Type":"ContainerStarted","Data":"256e938578634d0f6c40a4207db81f730184fa30c0e8a67e33ab6936caefb3f3"} Oct 29 22:20:09.238101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:09.238067 5367 generic.go:356] "Generic (PLEG): container finished" podID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerID="f938f86b99c85a1470fe0ac39023f93d472eb342ac1b00497fdc7fb957ed9d78" exitCode=0 Oct 29 22:20:09.238594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:09.238128 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" event={"ID":"0b1dfa46-a410-41da-9a32-feca4887d12f","Type":"ContainerDied","Data":"f938f86b99c85a1470fe0ac39023f93d472eb342ac1b00497fdc7fb957ed9d78"} Oct 29 22:20:10.245579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:10.245542 5367 generic.go:356] "Generic (PLEG): container finished" podID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerID="b0ec0f6665c834f99ee3e61b117c357413202fed43f10a031aebeb26518a7d39" exitCode=0 Oct 29 22:20:10.245995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:10.245626 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" event={"ID":"0b1dfa46-a410-41da-9a32-feca4887d12f","Type":"ContainerDied","Data":"b0ec0f6665c834f99ee3e61b117c357413202fed43f10a031aebeb26518a7d39"} Oct 29 22:20:11.370509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.370487 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:11.432179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.432151 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util\") pod \"0b1dfa46-a410-41da-9a32-feca4887d12f\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " Oct 29 22:20:11.432288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.432181 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle\") pod \"0b1dfa46-a410-41da-9a32-feca4887d12f\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " Oct 29 22:20:11.432288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.432201 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w\" (UniqueName: \"kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w\") pod \"0b1dfa46-a410-41da-9a32-feca4887d12f\" (UID: \"0b1dfa46-a410-41da-9a32-feca4887d12f\") " Oct 29 22:20:11.432840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.432808 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle" pod "0b1dfa46-a410-41da-9a32-feca4887d12f" (UID: "0b1dfa46-a410-41da-9a32-feca4887d12f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:11.434122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.434096 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w" pod "0b1dfa46-a410-41da-9a32-feca4887d12f" (UID: "0b1dfa46-a410-41da-9a32-feca4887d12f"). InnerVolumeSpecName "kube-api-access-nks4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:11.437662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.437555 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util" pod "0b1dfa46-a410-41da-9a32-feca4887d12f" (UID: "0b1dfa46-a410-41da-9a32-feca4887d12f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:11.533152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.533094 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:20:11.533152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.533116 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b1dfa46-a410-41da-9a32-feca4887d12f-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:20:11.533152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:11.533127 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nks4w\" (UniqueName: \"kubernetes.io/projected/0b1dfa46-a410-41da-9a32-feca4887d12f-kube-api-access-nks4w\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:20:12.255734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:12.255679 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" event={"ID":"0b1dfa46-a410-41da-9a32-feca4887d12f","Type":"ContainerDied","Data":"256e938578634d0f6c40a4207db81f730184fa30c0e8a67e33ab6936caefb3f3"} Oct 29 22:20:12.255908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:12.255747 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zrszl" Oct 29 22:20:12.255908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:12.255749 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256e938578634d0f6c40a4207db81f730184fa30c0e8a67e33ab6936caefb3f3" Oct 29 22:20:17.447610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447519 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd"] Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447841 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="extract" Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447853 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="extract" Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447865 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="pull" Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447870 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="pull" Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447896 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="util" Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447901 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="util" Oct 29 22:20:17.448021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.447951 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1dfa46-a410-41da-9a32-feca4887d12f" containerName="extract" Oct 29 22:20:17.449828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.449811 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.469418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.469392 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:20:17.469418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.469395 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:20:17.483484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.483456 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:20:17.483668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.483514 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kv9sm\"" Oct 29 22:20:17.483668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.483654 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:20:17.483777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.483758 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:20:17.502476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.502447 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd"] Oct 29 22:20:17.582218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.582181 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwxl\" (UniqueName: \"kubernetes.io/projected/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-kube-api-access-mdwxl\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.582218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.582222 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.582478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.582305 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.582478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.582407 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.582478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.582453 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.582593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.582491 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.683666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.683618 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwxl\" (UniqueName: \"kubernetes.io/projected/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-kube-api-access-mdwxl\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.683666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.683666 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.683921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.683817 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.683921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.683871 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.684012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.683989 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.684065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.684047 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.684153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.684133 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.686271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.686240 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.686418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.686381 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.686418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.686379 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.686542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.686518 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.708610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.708533 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwxl\" (UniqueName: \"kubernetes.io/projected/7d8ff79f-75a0-4983-a84c-2dbdb32e255b-kube-api-access-mdwxl\") pod \"opendatahub-operator-controller-manager-b95c7d58c-cv5kd\" (UID: \"7d8ff79f-75a0-4983-a84c-2dbdb32e255b\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:17.762748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:17.762688 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:18.111613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:18.111548 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd"] Oct 29 22:20:18.114419 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:20:18.114390 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8ff79f_75a0_4983_a84c_2dbdb32e255b.slice/crio-6295b700bf7e8b1dc5bca73763f60366a271ca8fbc3769458046a437d1cdc46e WatchSource:0}: Error finding container 6295b700bf7e8b1dc5bca73763f60366a271ca8fbc3769458046a437d1cdc46e: Status 404 returned error can't find the container with id 6295b700bf7e8b1dc5bca73763f60366a271ca8fbc3769458046a437d1cdc46e Oct 29 22:20:18.280663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:18.280621 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" event={"ID":"7d8ff79f-75a0-4983-a84c-2dbdb32e255b","Type":"ContainerStarted","Data":"6295b700bf7e8b1dc5bca73763f60366a271ca8fbc3769458046a437d1cdc46e"} Oct 29 22:20:21.295539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:21.295456 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" event={"ID":"7d8ff79f-75a0-4983-a84c-2dbdb32e255b","Type":"ContainerStarted","Data":"8e0614363f17b3e55aa243ee9d1755f7ee4d8c544908ab8b9617c64301bc8f5e"} Oct 29 22:20:22.301547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:22.301509 5367 generic.go:356] "Generic (PLEG): container finished" podID="7d8ff79f-75a0-4983-a84c-2dbdb32e255b" containerID="8e0614363f17b3e55aa243ee9d1755f7ee4d8c544908ab8b9617c64301bc8f5e" exitCode=0 Oct 29 22:20:22.301990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:22.301598 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" event={"ID":"7d8ff79f-75a0-4983-a84c-2dbdb32e255b","Type":"ContainerDied","Data":"8e0614363f17b3e55aa243ee9d1755f7ee4d8c544908ab8b9617c64301bc8f5e"} Oct 29 22:20:22.330365 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:20:22.330328 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:20:23.307646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:23.307612 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" event={"ID":"7d8ff79f-75a0-4983-a84c-2dbdb32e255b","Type":"ContainerStarted","Data":"7a1ee831d6133a6a32f6b74e21cddd0042ffa1119e36e45a75671bd3ab002f10"} Oct 29 22:20:23.308099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:23.307757 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:23.398414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:23.398362 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" podStartSLOduration=2.909354252 podStartE2EDuration="6.398348841s" totalImagesPullingTime="3.006895995s" totalInitContainerRuntime="482.098594ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:23.384172378 +0000 UTC m=+290.278247299" watchObservedRunningTime="2026-10-29 22:20:23.398348841 +0000 UTC m=+290.292423761" Oct 29 22:20:34.313987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:34.313954 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd" Oct 29 22:20:37.340101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.340066 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b649d648f-s7zqw"] Oct 29 22:20:37.347992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.347967 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.365182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.365148 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b649d648f-s7zqw"] Oct 29 22:20:37.452139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.452104 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:20:37.452331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.452180 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:20:37.452331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.452302 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 22:20:37.452445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.452369 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5k5vk\"" Oct 29 22:20:37.452445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.452373 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 22:20:37.454734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.454713 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvl6\" (UniqueName: \"kubernetes.io/projected/d46945c5-160d-485a-a63a-e90f7e2be9fc-kube-api-access-pvvl6\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.454796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.454782 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46945c5-160d-485a-a63a-e90f7e2be9fc-tmp\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.454876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.454860 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d46945c5-160d-485a-a63a-e90f7e2be9fc-tls-certs\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.556163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.556125 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46945c5-160d-485a-a63a-e90f7e2be9fc-tmp\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.556163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.556163 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d46945c5-160d-485a-a63a-e90f7e2be9fc-tls-certs\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.556367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.556204 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvl6\" (UniqueName: \"kubernetes.io/projected/d46945c5-160d-485a-a63a-e90f7e2be9fc-kube-api-access-pvvl6\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.558575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.558545 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46945c5-160d-485a-a63a-e90f7e2be9fc-tmp\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.558686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.558671 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d46945c5-160d-485a-a63a-e90f7e2be9fc-tls-certs\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.578689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.578664 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvl6\" (UniqueName: \"kubernetes.io/projected/d46945c5-160d-485a-a63a-e90f7e2be9fc-kube-api-access-pvvl6\") pod \"kube-auth-proxy-b649d648f-s7zqw\" (UID: \"d46945c5-160d-485a-a63a-e90f7e2be9fc\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.661165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.661093 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" Oct 29 22:20:37.972139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:37.972114 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b649d648f-s7zqw"] Oct 29 22:20:38.369960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:38.369926 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" event={"ID":"d46945c5-160d-485a-a63a-e90f7e2be9fc","Type":"ContainerStarted","Data":"e4bd873d78b4674208a8d4444f2b9bcd4cd55aca972fccbfacd6313ec5a93cdf"} Oct 29 22:20:39.522588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.522552 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r"] Oct 29 22:20:39.531209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.531186 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.546869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.546845 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r"] Oct 29 22:20:39.577037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.576768 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8454\" (UniqueName: \"kubernetes.io/projected/e4216cff-be15-4c5a-b701-5ad318a5ffe3-kube-api-access-d8454\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.577037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.576852 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4216cff-be15-4c5a-b701-5ad318a5ffe3-metrics-cert\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.577037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.576884 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4216cff-be15-4c5a-b701-5ad318a5ffe3-cert\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.577037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.576904 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e4216cff-be15-4c5a-b701-5ad318a5ffe3-manager-config\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.637866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.637836 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 29 22:20:39.638002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.637869 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 29 22:20:39.638002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.637939 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tjxgg\"" Oct 29 22:20:39.638106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.638013 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 29 22:20:39.677645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.677615 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8454\" (UniqueName: \"kubernetes.io/projected/e4216cff-be15-4c5a-b701-5ad318a5ffe3-kube-api-access-d8454\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.677876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.677671 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4216cff-be15-4c5a-b701-5ad318a5ffe3-metrics-cert\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.677876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.677721 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4216cff-be15-4c5a-b701-5ad318a5ffe3-cert\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.677876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.677743 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e4216cff-be15-4c5a-b701-5ad318a5ffe3-manager-config\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.678423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.678400 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e4216cff-be15-4c5a-b701-5ad318a5ffe3-manager-config\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.680348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.680326 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4216cff-be15-4c5a-b701-5ad318a5ffe3-cert\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.680428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.680376 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4216cff-be15-4c5a-b701-5ad318a5ffe3-metrics-cert\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.700218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.700189 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8454\" (UniqueName: \"kubernetes.io/projected/e4216cff-be15-4c5a-b701-5ad318a5ffe3-kube-api-access-d8454\") pod \"lws-controller-manager-76cd858f56-pqg6r\" (UID: \"e4216cff-be15-4c5a-b701-5ad318a5ffe3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:39.846587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:39.846553 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:40.146550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:40.146470 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r"] Oct 29 22:20:40.150223 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:20:40.150195 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4216cff_be15_4c5a_b701_5ad318a5ffe3.slice/crio-262437e50a1a8f588f6efe2b99c406bebf3d5747a6f94319acb46aa607b0366c WatchSource:0}: Error finding container 262437e50a1a8f588f6efe2b99c406bebf3d5747a6f94319acb46aa607b0366c: Status 404 returned error can't find the container with id 262437e50a1a8f588f6efe2b99c406bebf3d5747a6f94319acb46aa607b0366c Oct 29 22:20:40.379914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:40.379877 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" event={"ID":"d46945c5-160d-485a-a63a-e90f7e2be9fc","Type":"ContainerStarted","Data":"e2cfbfa308ed522d90503ead687976fc43142d1925484637f55ec6e8bd55b194"} Oct 29 22:20:40.381091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:40.381065 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" event={"ID":"e4216cff-be15-4c5a-b701-5ad318a5ffe3","Type":"ContainerStarted","Data":"262437e50a1a8f588f6efe2b99c406bebf3d5747a6f94319acb46aa607b0366c"} Oct 29 22:20:40.425607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:40.425521 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b649d648f-s7zqw" podStartSLOduration=1.206704671 podStartE2EDuration="3.425507733s" totalImagesPullingTime="2.218803062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:40.424350487 +0000 UTC m=+307.318425409" watchObservedRunningTime="2026-10-29 22:20:40.425507733 +0000 UTC m=+307.319582668" Oct 29 22:20:42.390347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:42.390310 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" event={"ID":"e4216cff-be15-4c5a-b701-5ad318a5ffe3","Type":"ContainerStarted","Data":"37515746429dbf8306bdfa6443df10124b2d1895211c0e9ffe2d75234ad4728c"} Oct 29 22:20:42.390741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:42.390391 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:20:42.433041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:42.432999 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" podStartSLOduration=1.905909442 podStartE2EDuration="3.432986578s" totalImagesPullingTime="1.527077136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:42.432603488 +0000 UTC m=+309.326678410" watchObservedRunningTime="2026-10-29 22:20:42.432986578 +0000 UTC m=+309.327061498" Oct 29 22:20:52.330280 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:20:52.330235 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:20:53.396430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:20:53.396399 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-pqg6r" Oct 29 22:21:22.329257 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:21:22.329220 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:21:52.330252 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:21:52.330170 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:22:22.330207 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:22:22.330169 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:22:32.016277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.016241 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6"] Oct 29 22:22:32.018926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.018909 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.049343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.049305 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rp4xx\"" Oct 29 22:22:32.049343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.049314 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 22:22:32.049533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.049339 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 22:22:32.050710 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.050673 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6"] Oct 29 22:22:32.092665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.092637 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.092842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.092727 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.092842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.092767 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2bz\" (UniqueName: \"kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.117137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.117109 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4"] Oct 29 22:22:32.119787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.119772 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.135853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.135830 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-dr544\"" Oct 29 22:22:32.141101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.141074 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4"] Oct 29 22:22:32.193910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.193878 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.194083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.193919 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.194083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.194021 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2bz\" (UniqueName: \"kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.194083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.194059 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspdh\" (UniqueName: \"kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.194212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.194084 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.194212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.194186 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.194376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.194350 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.194484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.194469 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.217236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.217205 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2bz\" (UniqueName: \"kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.294638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.294543 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.294638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.294613 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.294917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.294782 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspdh\" (UniqueName: \"kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.294999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.294981 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.295060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.295042 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.320177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.320146 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspdh\" (UniqueName: \"kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.331543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.331519 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:32.333361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.333338 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh"] Oct 29 22:22:32.335739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.335713 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.353554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.353516 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-mn55x\"" Oct 29 22:22:32.358871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.358847 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh"] Oct 29 22:22:32.387341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.387311 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh"] Oct 29 22:22:32.396380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.395770 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.396380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.395954 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbmw\" (UniqueName: \"kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.396380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.395982 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.396380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.396014 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.413334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.413307 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh"] Oct 29 22:22:32.413334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.413338 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fgv7w\"" Oct 29 22:22:32.432994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.432973 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:32.497382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497358 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.497496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497394 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbmw\" (UniqueName: \"kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.497496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497416 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.497572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497545 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.497620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497589 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.497780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497756 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcdb\" (UniqueName: \"kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.498900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.497869 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.498900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.498343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.518985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.518965 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbmw\" (UniqueName: \"kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.599215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.599186 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.599356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.599224 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcdb\" (UniqueName: \"kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.599356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.599271 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.599635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.599610 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.599681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.599658 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.620300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.620269 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcdb\" (UniqueName: \"kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.625839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.625818 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6"] Oct 29 22:22:32.628268 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:22:32.628246 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf019a14_7cfb_4e13_b3cb_700e327006ba.slice/crio-5e4d3586b2f6405a5f67634f12e7b05b7b853a402534902f519499583ee6ed3a WatchSource:0}: Error finding container 5e4d3586b2f6405a5f67634f12e7b05b7b853a402534902f519499583ee6ed3a: Status 404 returned error can't find the container with id 5e4d3586b2f6405a5f67634f12e7b05b7b853a402534902f519499583ee6ed3a Oct 29 22:22:32.663181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.663157 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:32.709799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.709677 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:32.726553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.726377 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4"] Oct 29 22:22:32.728923 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:22:32.728888 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210b05c5_f744_43dd_b2b1_ec7f4d8d3f11.slice/crio-698a7ede58992260fed78fdaef1c6fe02d7bf07e03137e7618abf69b5630d0d4 WatchSource:0}: Error finding container 698a7ede58992260fed78fdaef1c6fe02d7bf07e03137e7618abf69b5630d0d4: Status 404 returned error can't find the container with id 698a7ede58992260fed78fdaef1c6fe02d7bf07e03137e7618abf69b5630d0d4 Oct 29 22:22:32.836475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.836441 5367 generic.go:356] "Generic (PLEG): container finished" podID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerID="ef23a8a0a79fae81de284b3be690be93e05e83f3104f1aec6d663fc4f0d955ae" exitCode=0 Oct 29 22:22:32.836656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.836524 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" event={"ID":"df019a14-7cfb-4e13-b3cb-700e327006ba","Type":"ContainerDied","Data":"ef23a8a0a79fae81de284b3be690be93e05e83f3104f1aec6d663fc4f0d955ae"} Oct 29 22:22:32.836656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.836576 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" event={"ID":"df019a14-7cfb-4e13-b3cb-700e327006ba","Type":"ContainerStarted","Data":"5e4d3586b2f6405a5f67634f12e7b05b7b853a402534902f519499583ee6ed3a"} Oct 29 22:22:32.837814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.837794 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:22:32.838353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.838331 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" event={"ID":"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11","Type":"ContainerStarted","Data":"8e2aae06c9ff87371e7933743ce26a780ead1803da177b9da48e01b1d7d6f577"} Oct 29 22:22:32.838432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.838381 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" event={"ID":"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11","Type":"ContainerStarted","Data":"698a7ede58992260fed78fdaef1c6fe02d7bf07e03137e7618abf69b5630d0d4"} Oct 29 22:22:32.958670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:32.958514 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh"] Oct 29 22:22:32.961239 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:22:32.961210 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23ab4b1_5ff4_4c8b_badf_28edf7315909.slice/crio-ca70e2743679823b3b6ac4b1eada6eea18f584d1a8c1c06c220d20c5eb43a224 WatchSource:0}: Error finding container ca70e2743679823b3b6ac4b1eada6eea18f584d1a8c1c06c220d20c5eb43a224: Status 404 returned error can't find the container with id ca70e2743679823b3b6ac4b1eada6eea18f584d1a8c1c06c220d20c5eb43a224 Oct 29 22:22:33.012474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.012447 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh"] Oct 29 22:22:33.015455 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:22:33.015430 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dd20a6_a0f8_4076_9b08_2cad30cb66f8.slice/crio-2671c8ac9338760288741cf67ec726808e726b2777137a294a6529a3918da399 WatchSource:0}: Error finding container 2671c8ac9338760288741cf67ec726808e726b2777137a294a6529a3918da399: Status 404 returned error can't find the container with id 2671c8ac9338760288741cf67ec726808e726b2777137a294a6529a3918da399 Oct 29 22:22:33.844627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.844594 5367 generic.go:356] "Generic (PLEG): container finished" podID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerID="4df770a328edacc7a63d3d47e90adea5ccdd34e93ec10f1d538f6997f3fa6c49" exitCode=0 Oct 29 22:22:33.844996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.844653 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" event={"ID":"df019a14-7cfb-4e13-b3cb-700e327006ba","Type":"ContainerDied","Data":"4df770a328edacc7a63d3d47e90adea5ccdd34e93ec10f1d538f6997f3fa6c49"} Oct 29 22:22:33.846348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.846235 5367 generic.go:356] "Generic (PLEG): container finished" podID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerID="8e2aae06c9ff87371e7933743ce26a780ead1803da177b9da48e01b1d7d6f577" exitCode=0 Oct 29 22:22:33.846348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.846304 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" event={"ID":"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11","Type":"ContainerDied","Data":"8e2aae06c9ff87371e7933743ce26a780ead1803da177b9da48e01b1d7d6f577"} Oct 29 22:22:33.847907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.847887 5367 generic.go:356] "Generic (PLEG): container finished" podID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerID="b9c31208269a041c85eee8bc9f5d8b1d3a86592a69e4ca24763589cef250ca5b" exitCode=0 Oct 29 22:22:33.847997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.847974 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" event={"ID":"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8","Type":"ContainerDied","Data":"b9c31208269a041c85eee8bc9f5d8b1d3a86592a69e4ca24763589cef250ca5b"} Oct 29 22:22:33.848078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.848022 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" event={"ID":"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8","Type":"ContainerStarted","Data":"2671c8ac9338760288741cf67ec726808e726b2777137a294a6529a3918da399"} Oct 29 22:22:33.850237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.850218 5367 generic.go:356] "Generic (PLEG): container finished" podID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerID="40b6d9d7ec56d1e6b5dcd7439329192222c9c826b8067908d3bb41e25dc46145" exitCode=0 Oct 29 22:22:33.850351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.850242 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" event={"ID":"b23ab4b1-5ff4-4c8b-badf-28edf7315909","Type":"ContainerDied","Data":"40b6d9d7ec56d1e6b5dcd7439329192222c9c826b8067908d3bb41e25dc46145"} Oct 29 22:22:33.850351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:33.850274 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" event={"ID":"b23ab4b1-5ff4-4c8b-badf-28edf7315909","Type":"ContainerStarted","Data":"ca70e2743679823b3b6ac4b1eada6eea18f584d1a8c1c06c220d20c5eb43a224"} Oct 29 22:22:34.856821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:34.856793 5367 generic.go:356] "Generic (PLEG): container finished" podID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerID="b93f21bc750b8f3903faa561d47c56911ed3fe604289b064998c695f262ed725" exitCode=0 Oct 29 22:22:34.857127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:34.856874 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" event={"ID":"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8","Type":"ContainerDied","Data":"b93f21bc750b8f3903faa561d47c56911ed3fe604289b064998c695f262ed725"} Oct 29 22:22:34.858832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:34.858808 5367 generic.go:356] "Generic (PLEG): container finished" podID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerID="e31120251531e7a1b39c53696af3378229f0478557ce2dab7cf0367d037eb541" exitCode=0 Oct 29 22:22:34.858914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:34.858892 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" event={"ID":"b23ab4b1-5ff4-4c8b-badf-28edf7315909","Type":"ContainerDied","Data":"e31120251531e7a1b39c53696af3378229f0478557ce2dab7cf0367d037eb541"} Oct 29 22:22:34.861374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:34.861346 5367 generic.go:356] "Generic (PLEG): container finished" podID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerID="4080b150791a6bf06ac9187b7c8bda427300ff1760d6b2f3240b5dc564ed9907" exitCode=0 Oct 29 22:22:34.861471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:34.861417 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" event={"ID":"df019a14-7cfb-4e13-b3cb-700e327006ba","Type":"ContainerDied","Data":"4080b150791a6bf06ac9187b7c8bda427300ff1760d6b2f3240b5dc564ed9907"} Oct 29 22:22:35.868416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:35.868379 5367 generic.go:356] "Generic (PLEG): container finished" podID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerID="f1688e9f8a1c15a807fdff51ffb437841aec3164de7b416b5d198c114b72ebd3" exitCode=0 Oct 29 22:22:35.868863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:35.868470 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" event={"ID":"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8","Type":"ContainerDied","Data":"f1688e9f8a1c15a807fdff51ffb437841aec3164de7b416b5d198c114b72ebd3"} Oct 29 22:22:35.870500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:35.870480 5367 generic.go:356] "Generic (PLEG): container finished" podID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerID="6f036cbe6461c536700a3e5cd9dd4877579248985267b3f4b6967ec8882e120e" exitCode=0 Oct 29 22:22:35.870581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:35.870544 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" event={"ID":"b23ab4b1-5ff4-4c8b-badf-28edf7315909","Type":"ContainerDied","Data":"6f036cbe6461c536700a3e5cd9dd4877579248985267b3f4b6967ec8882e120e"} Oct 29 22:22:35.872155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:35.872135 5367 generic.go:356] "Generic (PLEG): container finished" podID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerID="9d237d220f08e663d3434eeda575b553b36e28e5e5f49f2c0433be9c81389ad6" exitCode=0 Oct 29 22:22:35.872219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:35.872162 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" event={"ID":"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11","Type":"ContainerDied","Data":"9d237d220f08e663d3434eeda575b553b36e28e5e5f49f2c0433be9c81389ad6"} Oct 29 22:22:36.001491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.001467 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:36.024210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.024145 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util\") pod \"df019a14-7cfb-4e13-b3cb-700e327006ba\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " Oct 29 22:22:36.024210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.024212 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle\") pod \"df019a14-7cfb-4e13-b3cb-700e327006ba\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " Oct 29 22:22:36.024402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.024248 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz\" (UniqueName: \"kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz\") pod \"df019a14-7cfb-4e13-b3cb-700e327006ba\" (UID: \"df019a14-7cfb-4e13-b3cb-700e327006ba\") " Oct 29 22:22:36.025574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.025547 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle" pod "df019a14-7cfb-4e13-b3cb-700e327006ba" (UID: "df019a14-7cfb-4e13-b3cb-700e327006ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:36.026996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.026972 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz" pod "df019a14-7cfb-4e13-b3cb-700e327006ba" (UID: "df019a14-7cfb-4e13-b3cb-700e327006ba"). InnerVolumeSpecName "kube-api-access-4x2bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:36.030080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.030059 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util" pod "df019a14-7cfb-4e13-b3cb-700e327006ba" (UID: "df019a14-7cfb-4e13-b3cb-700e327006ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:36.124981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.124952 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:36.124981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.124977 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df019a14-7cfb-4e13-b3cb-700e327006ba-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:36.125121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.124986 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2bz\" (UniqueName: \"kubernetes.io/projected/df019a14-7cfb-4e13-b3cb-700e327006ba-kube-api-access-4x2bz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:36.879310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.879269 5367 generic.go:356] "Generic (PLEG): container finished" podID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerID="ecb4bd6366ba60c6fd0fd1445c24e4bd309514e8026b6ba80bc82139f5abfd46" exitCode=0 Oct 29 22:22:36.879685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.879344 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" event={"ID":"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11","Type":"ContainerDied","Data":"ecb4bd6366ba60c6fd0fd1445c24e4bd309514e8026b6ba80bc82139f5abfd46"} Oct 29 22:22:36.881213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.881185 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" event={"ID":"df019a14-7cfb-4e13-b3cb-700e327006ba","Type":"ContainerDied","Data":"5e4d3586b2f6405a5f67634f12e7b05b7b853a402534902f519499583ee6ed3a"} Oct 29 22:22:36.881304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.881220 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apzvl6" Oct 29 22:22:36.881304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:36.881231 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4d3586b2f6405a5f67634f12e7b05b7b853a402534902f519499583ee6ed3a" Oct 29 22:22:37.029219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.029158 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:37.035433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.035410 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:37.135556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.135530 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util\") pod \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " Oct 29 22:22:37.135740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.135564 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util\") pod \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " Oct 29 22:22:37.135740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.135589 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle\") pod \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " Oct 29 22:22:37.135740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.135629 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle\") pod \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " Oct 29 22:22:37.135740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.135648 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb\" (UniqueName: \"kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb\") pod \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\" (UID: \"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8\") " Oct 29 22:22:37.135740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.135665 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw\" (UniqueName: \"kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw\") pod \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\" (UID: \"b23ab4b1-5ff4-4c8b-badf-28edf7315909\") " Oct 29 22:22:37.136191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.136160 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle" pod "b23ab4b1-5ff4-4c8b-badf-28edf7315909" (UID: "b23ab4b1-5ff4-4c8b-badf-28edf7315909"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:37.136326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.136228 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle" pod "f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" (UID: "f5dd20a6-a0f8-4076-9b08-2cad30cb66f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:37.138257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.138219 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw" pod "b23ab4b1-5ff4-4c8b-badf-28edf7315909" (UID: "b23ab4b1-5ff4-4c8b-badf-28edf7315909"). InnerVolumeSpecName "kube-api-access-qpbmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:37.138355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.138275 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb" pod "f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" (UID: "f5dd20a6-a0f8-4076-9b08-2cad30cb66f8"). InnerVolumeSpecName "kube-api-access-kqcdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:37.142136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.142113 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util" pod "f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" (UID: "f5dd20a6-a0f8-4076-9b08-2cad30cb66f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:37.142338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.142317 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util" pod "b23ab4b1-5ff4-4c8b-badf-28edf7315909" (UID: "b23ab4b1-5ff4-4c8b-badf-28edf7315909"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:37.236918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.236897 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpbmw\" (UniqueName: \"kubernetes.io/projected/b23ab4b1-5ff4-4c8b-badf-28edf7315909-kube-api-access-qpbmw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:37.236918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.236918 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:37.237032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.236927 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:37.237032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.236935 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b23ab4b1-5ff4-4c8b-badf-28edf7315909-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:37.237032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.236943 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:37.237032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.236950 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcdb\" (UniqueName: \"kubernetes.io/projected/f5dd20a6-a0f8-4076-9b08-2cad30cb66f8-kube-api-access-kqcdb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:37.887516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.887485 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" Oct 29 22:22:37.887987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.887487 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tcthh" event={"ID":"f5dd20a6-a0f8-4076-9b08-2cad30cb66f8","Type":"ContainerDied","Data":"2671c8ac9338760288741cf67ec726808e726b2777137a294a6529a3918da399"} Oct 29 22:22:37.887987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.887608 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2671c8ac9338760288741cf67ec726808e726b2777137a294a6529a3918da399" Oct 29 22:22:37.889422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.889396 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" Oct 29 22:22:37.889528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.889422 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c689rkh" event={"ID":"b23ab4b1-5ff4-4c8b-badf-28edf7315909","Type":"ContainerDied","Data":"ca70e2743679823b3b6ac4b1eada6eea18f584d1a8c1c06c220d20c5eb43a224"} Oct 29 22:22:37.889528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:37.889466 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca70e2743679823b3b6ac4b1eada6eea18f584d1a8c1c06c220d20c5eb43a224" Oct 29 22:22:38.007430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.007404 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:38.045433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.045402 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh\" (UniqueName: \"kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh\") pod \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " Oct 29 22:22:38.045569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.045441 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle\") pod \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " Oct 29 22:22:38.045569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.045472 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util\") pod \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\" (UID: \"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11\") " Oct 29 22:22:38.045951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.045924 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle" pod "210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" (UID: "210b05c5-f744-43dd-b2b1-ec7f4d8d3f11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:38.047637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.047615 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh" pod "210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" (UID: "210b05c5-f744-43dd-b2b1-ec7f4d8d3f11"). InnerVolumeSpecName "kube-api-access-cspdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:38.050594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.050565 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util" pod "210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" (UID: "210b05c5-f744-43dd-b2b1-ec7f4d8d3f11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:38.146507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.146423 5367 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-util\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:38.146507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.146451 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspdh\" (UniqueName: \"kubernetes.io/projected/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-kube-api-access-cspdh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:38.146507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.146463 5367 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/210b05c5-f744-43dd-b2b1-ec7f4d8d3f11-bundle\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:22:38.896841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.896813 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" Oct 29 22:22:38.897188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.896838 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz6vp4" event={"ID":"210b05c5-f744-43dd-b2b1-ec7f4d8d3f11","Type":"ContainerDied","Data":"698a7ede58992260fed78fdaef1c6fe02d7bf07e03137e7618abf69b5630d0d4"} Oct 29 22:22:38.897188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:38.896885 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="698a7ede58992260fed78fdaef1c6fe02d7bf07e03137e7618abf69b5630d0d4" Oct 29 22:22:52.329644 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:22:52.329609 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:22:59.752380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752348 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx"] Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752781 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="pull" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752798 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="pull" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752818 5367 cpu_manager.go:414] "containerMap: removing container" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="util" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752824 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="util" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752833 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="extract" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752840 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="extract" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752848 5367 cpu_manager.go:414] "containerMap: removing container" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="pull" Oct 29 22:22:59.752857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752855 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="pull" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752868 5367 cpu_manager.go:414] "containerMap: removing container" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752875 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752891 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="util" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752899 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="util" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752908 5367 cpu_manager.go:414] "containerMap: removing container" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752915 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752929 5367 cpu_manager.go:414] "containerMap: removing container" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="pull" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752935 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="pull" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752947 5367 cpu_manager.go:414] "containerMap: removing container" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="util" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752954 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="util" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752966 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="pull" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752972 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="pull" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752987 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="util" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.752994 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="util" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.753013 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.753020 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.753091 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dd20a6-a0f8-4076-9b08-2cad30cb66f8" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.753102 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="df019a14-7cfb-4e13-b3cb-700e327006ba" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.753113 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="210b05c5-f744-43dd-b2b1-ec7f4d8d3f11" containerName="extract" Oct 29 22:22:59.753225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.753127 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23ab4b1-5ff4-4c8b-badf-28edf7315909" containerName="extract" Oct 29 22:22:59.756167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.756147 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.774546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.774523 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx"] Oct 29 22:22:59.786631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.786612 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 29 22:22:59.786738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.786639 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:22:59.786902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.786886 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:22:59.786968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.786913 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 29 22:22:59.787005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.786965 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kzfft\"" Oct 29 22:22:59.823861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.823842 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e03eaa49-3506-46db-ae0f-6df0e4973111-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.823960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.823890 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc79\" (UniqueName: \"kubernetes.io/projected/e03eaa49-3506-46db-ae0f-6df0e4973111-kube-api-access-vfc79\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.824002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.823969 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e03eaa49-3506-46db-ae0f-6df0e4973111-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.925199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.925174 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e03eaa49-3506-46db-ae0f-6df0e4973111-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.925339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.925224 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc79\" (UniqueName: \"kubernetes.io/projected/e03eaa49-3506-46db-ae0f-6df0e4973111-kube-api-access-vfc79\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.925339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.925244 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e03eaa49-3506-46db-ae0f-6df0e4973111-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.925818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.925802 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e03eaa49-3506-46db-ae0f-6df0e4973111-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.927667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.927647 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e03eaa49-3506-46db-ae0f-6df0e4973111-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:22:59.945769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:22:59.945747 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc79\" (UniqueName: \"kubernetes.io/projected/e03eaa49-3506-46db-ae0f-6df0e4973111-kube-api-access-vfc79\") pod \"kuadrant-console-plugin-66c4fb575d-2vqxx\" (UID: \"e03eaa49-3506-46db-ae0f-6df0e4973111\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:23:00.067556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:00.067531 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" Oct 29 22:23:00.359737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:00.359714 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx"] Oct 29 22:23:00.362330 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:23:00.362280 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode03eaa49_3506_46db_ae0f_6df0e4973111.slice/crio-5f151fe5fb0766da128c42f86c02e52ddf4c98d534f8f04471e1fac099ba02fe WatchSource:0}: Error finding container 5f151fe5fb0766da128c42f86c02e52ddf4c98d534f8f04471e1fac099ba02fe: Status 404 returned error can't find the container with id 5f151fe5fb0766da128c42f86c02e52ddf4c98d534f8f04471e1fac099ba02fe Oct 29 22:23:00.998386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:00.998348 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" event={"ID":"e03eaa49-3506-46db-ae0f-6df0e4973111","Type":"ContainerStarted","Data":"5f151fe5fb0766da128c42f86c02e52ddf4c98d534f8f04471e1fac099ba02fe"} Oct 29 22:23:05.021277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:05.021188 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" event={"ID":"e03eaa49-3506-46db-ae0f-6df0e4973111","Type":"ContainerStarted","Data":"87ab869fa291679ef15b3a101c0b16ec0072d139968f4d0ad231c1b611963cd2"} Oct 29 22:23:05.063268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:05.063223 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2vqxx" podStartSLOduration=1.818085876 podStartE2EDuration="6.063209734s" totalImagesPullingTime="4.245123858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:05.061982165 +0000 UTC m=+451.956057088" watchObservedRunningTime="2026-10-29 22:23:05.063209734 +0000 UTC m=+451.957284649" Oct 29 22:23:22.329800 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:23:22.329727 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:23:52.329050 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:23:52.329011 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:23:59.895965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:59.895930 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nk6m2"] Oct 29 22:23:59.898015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:59.897999 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:23:59.912554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:59.912536 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xmtnn\"" Oct 29 22:23:59.925861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:23:59.925832 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nk6m2"] Oct 29 22:24:00.010109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.010081 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vnn\" (UniqueName: \"kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn\") pod \"authorino-57f7f8659f-nk6m2\" (UID: \"9dd4d90e-7926-4875-abc0-2493125bb874\") " pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:24:00.110710 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.110672 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vnn\" (UniqueName: \"kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn\") pod \"authorino-57f7f8659f-nk6m2\" (UID: \"9dd4d90e-7926-4875-abc0-2493125bb874\") " pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:24:00.135066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.135036 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vnn\" (UniqueName: \"kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn\") pod \"authorino-57f7f8659f-nk6m2\" (UID: \"9dd4d90e-7926-4875-abc0-2493125bb874\") " pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:24:00.207762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.207688 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:24:00.389490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.389464 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2dw4c"] Oct 29 22:24:00.391795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.391780 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:00.412635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.412615 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2dw4c"] Oct 29 22:24:00.493719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.493685 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nk6m2"] Oct 29 22:24:00.496350 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:24:00.496312 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd4d90e_7926_4875_abc0_2493125bb874.slice/crio-28de6e7fa032ff16e237928a14dbdf4e60bdcd82612223b0b5eaa93504cce908 WatchSource:0}: Error finding container 28de6e7fa032ff16e237928a14dbdf4e60bdcd82612223b0b5eaa93504cce908: Status 404 returned error can't find the container with id 28de6e7fa032ff16e237928a14dbdf4e60bdcd82612223b0b5eaa93504cce908 Oct 29 22:24:00.514568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.514541 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ph77\" (UniqueName: \"kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77\") pod \"authorino-56c6c9ddd5-2dw4c\" (UID: \"4cc4d6ab-9594-4675-b732-ccc32de1baed\") " pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:00.615014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.614986 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ph77\" (UniqueName: \"kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77\") pod \"authorino-56c6c9ddd5-2dw4c\" (UID: \"4cc4d6ab-9594-4675-b732-ccc32de1baed\") " pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:00.634123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.634099 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ph77\" (UniqueName: \"kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77\") pod \"authorino-56c6c9ddd5-2dw4c\" (UID: \"4cc4d6ab-9594-4675-b732-ccc32de1baed\") " pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:00.701295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.701270 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:00.987547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:00.987519 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2dw4c"] Oct 29 22:24:00.989984 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:24:00.989960 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cc4d6ab_9594_4675_b732_ccc32de1baed.slice/crio-3b08791c4775a0ee65d09765570fc08a26d0e27e3709d52b1cccd35b215be04e WatchSource:0}: Error finding container 3b08791c4775a0ee65d09765570fc08a26d0e27e3709d52b1cccd35b215be04e: Status 404 returned error can't find the container with id 3b08791c4775a0ee65d09765570fc08a26d0e27e3709d52b1cccd35b215be04e Oct 29 22:24:01.267369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:01.267334 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" event={"ID":"9dd4d90e-7926-4875-abc0-2493125bb874","Type":"ContainerStarted","Data":"28de6e7fa032ff16e237928a14dbdf4e60bdcd82612223b0b5eaa93504cce908"} Oct 29 22:24:01.268478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:01.268454 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" event={"ID":"4cc4d6ab-9594-4675-b732-ccc32de1baed","Type":"ContainerStarted","Data":"3b08791c4775a0ee65d09765570fc08a26d0e27e3709d52b1cccd35b215be04e"} Oct 29 22:24:04.286650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:04.286614 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" event={"ID":"9dd4d90e-7926-4875-abc0-2493125bb874","Type":"ContainerStarted","Data":"c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8"} Oct 29 22:24:04.288190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:04.288167 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" event={"ID":"4cc4d6ab-9594-4675-b732-ccc32de1baed","Type":"ContainerStarted","Data":"72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179"} Oct 29 22:24:04.324710 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:04.324665 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" podStartSLOduration=2.305389461 podStartE2EDuration="5.324653356s" totalImagesPullingTime="3.019263895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:04.322811312 +0000 UTC m=+511.216886231" watchObservedRunningTime="2026-10-29 22:24:04.324653356 +0000 UTC m=+511.218728267" Oct 29 22:24:04.359464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:04.359417 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" podStartSLOduration=1.874461664 podStartE2EDuration="4.359404122s" totalImagesPullingTime="2.484942458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:04.358764274 +0000 UTC m=+511.252839194" watchObservedRunningTime="2026-10-29 22:24:04.359404122 +0000 UTC m=+511.253479071" Oct 29 22:24:04.383746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:04.383721 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nk6m2"] Oct 29 22:24:06.295646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:06.295609 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" podUID="9dd4d90e-7926-4875-abc0-2493125bb874" containerName="authorino" containerID="cri-o://c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8" gracePeriod=30 Oct 29 22:24:06.534974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:06.534950 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:24:06.667249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:06.667216 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn\" (UniqueName: \"kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn\") pod \"9dd4d90e-7926-4875-abc0-2493125bb874\" (UID: \"9dd4d90e-7926-4875-abc0-2493125bb874\") " Oct 29 22:24:06.669357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:06.669333 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn" pod "9dd4d90e-7926-4875-abc0-2493125bb874" (UID: "9dd4d90e-7926-4875-abc0-2493125bb874"). InnerVolumeSpecName "kube-api-access-h4vnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:06.768342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:06.768314 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vnn\" (UniqueName: \"kubernetes.io/projected/9dd4d90e-7926-4875-abc0-2493125bb874-kube-api-access-h4vnn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:24:07.301521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.301486 5367 generic.go:356] "Generic (PLEG): container finished" podID="9dd4d90e-7926-4875-abc0-2493125bb874" containerID="c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8" exitCode=0 Oct 29 22:24:07.301943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.301525 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" Oct 29 22:24:07.301943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.301536 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" event={"ID":"9dd4d90e-7926-4875-abc0-2493125bb874","Type":"ContainerDied","Data":"c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8"} Oct 29 22:24:07.301943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.301564 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nk6m2" event={"ID":"9dd4d90e-7926-4875-abc0-2493125bb874","Type":"ContainerDied","Data":"28de6e7fa032ff16e237928a14dbdf4e60bdcd82612223b0b5eaa93504cce908"} Oct 29 22:24:07.301943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.301580 5367 scope.go:122] "RemoveContainer" containerID="c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8" Oct 29 22:24:07.309643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.309625 5367 scope.go:122] "RemoveContainer" containerID="c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8" Oct 29 22:24:07.309926 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:24:07.309907 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8\": container with ID starting with c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8 not found: ID does not exist" containerID="c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8" Oct 29 22:24:07.309994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.309938 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8"} err="failed to get container status \"c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8\": rpc error: code = NotFound desc = could not find container \"c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8\": container with ID starting with c9fc16911a83a5b4b87785a00cfdb581da6c668a5dd0a36c02fbf4b6ef9059f8 not found: ID does not exist" Oct 29 22:24:07.349390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.349367 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nk6m2"] Oct 29 22:24:07.352988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:07.352971 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nk6m2"] Oct 29 22:24:09.219268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:09.219235 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd4d90e-7926-4875-abc0-2493125bb874" path="/var/lib/kubelet/pods/9dd4d90e-7926-4875-abc0-2493125bb874/volumes" Oct 29 22:24:22.328939 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:24:22.328899 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:24:34.446748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.446716 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56x9t"] Oct 29 22:24:34.447148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.447019 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9dd4d90e-7926-4875-abc0-2493125bb874" containerName="authorino" Oct 29 22:24:34.447148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.447029 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd4d90e-7926-4875-abc0-2493125bb874" containerName="authorino" Oct 29 22:24:34.447148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.447097 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd4d90e-7926-4875-abc0-2493125bb874" containerName="authorino" Oct 29 22:24:34.451189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.451172 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:34.469632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.469608 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56x9t"] Oct 29 22:24:34.581315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.581279 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkkc\" (UniqueName: \"kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc\") pod \"authorino-6f6944bcdd-56x9t\" (UID: \"cb6a9df1-98fb-4619-9c67-779203bb81b2\") " pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:34.629795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.629765 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56x9t"] Oct 29 22:24:34.629967 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:24:34.629950 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6hkkc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-56x9t" podUID="cb6a9df1-98fb-4619-9c67-779203bb81b2" Oct 29 22:24:34.653175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.653149 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2qlwv"] Oct 29 22:24:34.656314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.656297 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.674489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.674466 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 29 22:24:34.676146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.676126 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2qlwv"] Oct 29 22:24:34.681728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.681690 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkkc\" (UniqueName: \"kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc\") pod \"authorino-6f6944bcdd-56x9t\" (UID: \"cb6a9df1-98fb-4619-9c67-779203bb81b2\") " pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:34.702074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.702023 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkkc\" (UniqueName: \"kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc\") pod \"authorino-6f6944bcdd-56x9t\" (UID: \"cb6a9df1-98fb-4619-9c67-779203bb81b2\") " pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:34.782820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.782790 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjgl\" (UniqueName: \"kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl\") pod \"authorino-6889cbd8c4-2qlwv\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.782947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.782833 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert\") pod \"authorino-6889cbd8c4-2qlwv\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.800212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.800187 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2qlwv"] Oct 29 22:24:34.800363 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:24:34.800347 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9qjgl tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" podUID="83985625-051a-45ee-9f0d-da4802bed3f5" Oct 29 22:24:34.826786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.826761 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cf6f85954-95tgr"] Oct 29 22:24:34.829938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.829924 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:34.851227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.851206 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cf6f85954-95tgr"] Oct 29 22:24:34.883418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.883374 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjgl\" (UniqueName: \"kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl\") pod \"authorino-6889cbd8c4-2qlwv\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.883533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.883466 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert\") pod \"authorino-6889cbd8c4-2qlwv\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.883690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.883669 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xlrr\" (UniqueName: \"kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr\") pod \"authorino-5cf6f85954-95tgr\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:34.883780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.883736 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert\") pod \"authorino-5cf6f85954-95tgr\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:34.885802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.885781 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert\") pod \"authorino-6889cbd8c4-2qlwv\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.912684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.912658 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjgl\" (UniqueName: \"kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl\") pod \"authorino-6889cbd8c4-2qlwv\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:34.984879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.984824 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xlrr\" (UniqueName: \"kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr\") pod \"authorino-5cf6f85954-95tgr\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:34.984879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.984857 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert\") pod \"authorino-5cf6f85954-95tgr\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:34.986834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:34.986817 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert\") pod \"authorino-5cf6f85954-95tgr\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:35.004694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.004673 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xlrr\" (UniqueName: \"kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr\") pod \"authorino-5cf6f85954-95tgr\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:35.141365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.141332 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:24:35.427556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.427519 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:35.427738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.427519 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:35.430310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.430284 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cf6f85954-95tgr"] Oct 29 22:24:35.433948 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:24:35.433925 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bede3ca_7df4_445f_9b88_7cf08843ec2a.slice/crio-e01e4f3be9585e14c843bc486105470fa54e558ad2e7cbcbb9f920df76edc3f3 WatchSource:0}: Error finding container e01e4f3be9585e14c843bc486105470fa54e558ad2e7cbcbb9f920df76edc3f3: Status 404 returned error can't find the container with id e01e4f3be9585e14c843bc486105470fa54e558ad2e7cbcbb9f920df76edc3f3 Oct 29 22:24:35.436733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.436693 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:35.438811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.438794 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:35.488479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.488454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert\" (UniqueName: \"kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert\") pod \"83985625-051a-45ee-9f0d-da4802bed3f5\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " Oct 29 22:24:35.488835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.488525 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc\" (UniqueName: \"kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc\") pod \"cb6a9df1-98fb-4619-9c67-779203bb81b2\" (UID: \"cb6a9df1-98fb-4619-9c67-779203bb81b2\") " Oct 29 22:24:35.488835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.488548 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl\" (UniqueName: \"kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl\") pod \"83985625-051a-45ee-9f0d-da4802bed3f5\" (UID: \"83985625-051a-45ee-9f0d-da4802bed3f5\") " Oct 29 22:24:35.490577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.490551 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc" pod "cb6a9df1-98fb-4619-9c67-779203bb81b2" (UID: "cb6a9df1-98fb-4619-9c67-779203bb81b2"). InnerVolumeSpecName "kube-api-access-6hkkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:35.490577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.490562 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl" pod "83985625-051a-45ee-9f0d-da4802bed3f5" (UID: "83985625-051a-45ee-9f0d-da4802bed3f5"). InnerVolumeSpecName "kube-api-access-9qjgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:35.490577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.490552 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert" pod "83985625-051a-45ee-9f0d-da4802bed3f5" (UID: "83985625-051a-45ee-9f0d-da4802bed3f5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:24:35.589136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.589102 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkkc\" (UniqueName: \"kubernetes.io/projected/cb6a9df1-98fb-4619-9c67-779203bb81b2-kube-api-access-6hkkc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:24:35.589136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.589132 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjgl\" (UniqueName: \"kubernetes.io/projected/83985625-051a-45ee-9f0d-da4802bed3f5-kube-api-access-9qjgl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:24:35.589316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:35.589151 5367 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83985625-051a-45ee-9f0d-da4802bed3f5-tls-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:24:36.433422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.433388 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cf6f85954-95tgr" event={"ID":"0bede3ca-7df4-445f-9b88-7cf08843ec2a","Type":"ContainerStarted","Data":"12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a"} Oct 29 22:24:36.433422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.433408 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2qlwv" Oct 29 22:24:36.433422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.433418 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56x9t" Oct 29 22:24:36.433743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.433431 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cf6f85954-95tgr" event={"ID":"0bede3ca-7df4-445f-9b88-7cf08843ec2a","Type":"ContainerStarted","Data":"e01e4f3be9585e14c843bc486105470fa54e558ad2e7cbcbb9f920df76edc3f3"} Oct 29 22:24:36.488618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.488577 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5cf6f85954-95tgr" podStartSLOduration=1.909016524 podStartE2EDuration="2.488564647s" totalImagesPullingTime="579.548123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:36.486620377 +0000 UTC m=+543.380695293" watchObservedRunningTime="2026-10-29 22:24:36.488564647 +0000 UTC m=+543.382639563" Oct 29 22:24:36.516046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.516019 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2dw4c"] Oct 29 22:24:36.516223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.516203 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" podUID="4cc4d6ab-9594-4675-b732-ccc32de1baed" containerName="authorino" containerID="cri-o://72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179" gracePeriod=30 Oct 29 22:24:36.544881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.544857 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56x9t"] Oct 29 22:24:36.550192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.550169 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56x9t"] Oct 29 22:24:36.612334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.612298 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2qlwv"] Oct 29 22:24:36.617113 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.617072 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2qlwv"] Oct 29 22:24:36.775792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.775764 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:36.898835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.898803 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77\" (UniqueName: \"kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77\") pod \"4cc4d6ab-9594-4675-b732-ccc32de1baed\" (UID: \"4cc4d6ab-9594-4675-b732-ccc32de1baed\") " Oct 29 22:24:36.900754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.900728 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77" pod "4cc4d6ab-9594-4675-b732-ccc32de1baed" (UID: "4cc4d6ab-9594-4675-b732-ccc32de1baed"). InnerVolumeSpecName "kube-api-access-5ph77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:36.999980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:36.999894 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ph77\" (UniqueName: \"kubernetes.io/projected/4cc4d6ab-9594-4675-b732-ccc32de1baed-kube-api-access-5ph77\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:24:37.219145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.219114 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83985625-051a-45ee-9f0d-da4802bed3f5" path="/var/lib/kubelet/pods/83985625-051a-45ee-9f0d-da4802bed3f5/volumes" Oct 29 22:24:37.219335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.219323 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6a9df1-98fb-4619-9c67-779203bb81b2" path="/var/lib/kubelet/pods/cb6a9df1-98fb-4619-9c67-779203bb81b2/volumes" Oct 29 22:24:37.438759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.438727 5367 generic.go:356] "Generic (PLEG): container finished" podID="4cc4d6ab-9594-4675-b732-ccc32de1baed" containerID="72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179" exitCode=0 Oct 29 22:24:37.438970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.438759 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" event={"ID":"4cc4d6ab-9594-4675-b732-ccc32de1baed","Type":"ContainerDied","Data":"72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179"} Oct 29 22:24:37.438970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.438776 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" Oct 29 22:24:37.438970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.438802 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2dw4c" event={"ID":"4cc4d6ab-9594-4675-b732-ccc32de1baed","Type":"ContainerDied","Data":"3b08791c4775a0ee65d09765570fc08a26d0e27e3709d52b1cccd35b215be04e"} Oct 29 22:24:37.438970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.438819 5367 scope.go:122] "RemoveContainer" containerID="72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179" Oct 29 22:24:37.447014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.446994 5367 scope.go:122] "RemoveContainer" containerID="72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179" Oct 29 22:24:37.447290 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:24:37.447273 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179\": container with ID starting with 72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179 not found: ID does not exist" containerID="72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179" Oct 29 22:24:37.447349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.447299 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179"} err="failed to get container status \"72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179\": rpc error: code = NotFound desc = could not find container \"72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179\": container with ID starting with 72869ef9176476d141f0b656cb329140c34331a15d48d69646060804bceee179 not found: ID does not exist" Oct 29 22:24:37.492208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.492175 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2dw4c"] Oct 29 22:24:37.505211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:37.505182 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2dw4c"] Oct 29 22:24:39.218463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:24:39.218429 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc4d6ab-9594-4675-b732-ccc32de1baed" path="/var/lib/kubelet/pods/4cc4d6ab-9594-4675-b732-ccc32de1baed/volumes" Oct 29 22:24:52.329815 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:24:52.329741 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:25:22.329063 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:25:22.329030 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:25:52.329010 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:25:52.328971 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:26:21.616210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.616130 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np"] Oct 29 22:26:21.616647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.616495 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4cc4d6ab-9594-4675-b732-ccc32de1baed" containerName="authorino" Oct 29 22:26:21.616647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.616511 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc4d6ab-9594-4675-b732-ccc32de1baed" containerName="authorino" Oct 29 22:26:21.616647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.616603 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc4d6ab-9594-4675-b732-ccc32de1baed" containerName="authorino" Oct 29 22:26:21.622418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.622367 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.641320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.641289 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np"] Oct 29 22:26:21.641884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.641861 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 22:26:21.652514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.652484 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 22:26:21.652514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.652484 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 29 22:26:21.652514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.652484 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-59nr7\"" Oct 29 22:26:21.653680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.653659 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.653761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.653691 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.653761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.653732 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dfa063-b32f-46d7-975a-378ad1d59be9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.653888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.653865 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.653964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.653948 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487xj\" (UniqueName: \"kubernetes.io/projected/d4dfa063-b32f-46d7-975a-378ad1d59be9-kube-api-access-487xj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.654019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.653983 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.754977 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755025 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dfa063-b32f-46d7-975a-378ad1d59be9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755140 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755202 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487xj\" (UniqueName: \"kubernetes.io/projected/d4dfa063-b32f-46d7-975a-378ad1d59be9-kube-api-access-487xj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755224 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755260 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755472 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755566 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.755659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.755642 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.757438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.757416 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dfa063-b32f-46d7-975a-378ad1d59be9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.757613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.757595 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dfa063-b32f-46d7-975a-378ad1d59be9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.776164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.776125 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487xj\" (UniqueName: \"kubernetes.io/projected/d4dfa063-b32f-46d7-975a-378ad1d59be9-kube-api-access-487xj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zz4np\" (UID: \"d4dfa063-b32f-46d7-975a-378ad1d59be9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:21.936331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:21.936243 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:22.231784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:22.231759 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np"] Oct 29 22:26:22.234173 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:22.234143 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dfa063_b32f_46d7_975a_378ad1d59be9.slice/crio-b8148f6063f73de9709fc8347f59237daf6f476f46904c318e218b823769fc98 WatchSource:0}: Error finding container b8148f6063f73de9709fc8347f59237daf6f476f46904c318e218b823769fc98: Status 404 returned error can't find the container with id b8148f6063f73de9709fc8347f59237daf6f476f46904c318e218b823769fc98 Oct 29 22:26:22.330495 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:26:22.330455 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:26:22.905538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:22.905499 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" event={"ID":"d4dfa063-b32f-46d7-975a-378ad1d59be9","Type":"ContainerStarted","Data":"b8148f6063f73de9709fc8347f59237daf6f476f46904c318e218b823769fc98"} Oct 29 22:26:23.536991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.536960 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn"] Oct 29 22:26:23.540487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.540452 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.557381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.557353 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 29 22:26:23.572217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572186 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn"] Oct 29 22:26:23.572489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572461 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.572534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572490 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.572534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572512 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.572724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572686 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f18feb0-835d-45e8-a8e2-0053321ce57a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.572814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572797 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.572856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.572840 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/8f18feb0-835d-45e8-a8e2-0053321ce57a-kube-api-access-p899n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.673950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.673863 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f18feb0-835d-45e8-a8e2-0053321ce57a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.673950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.673920 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.673950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.673942 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/8f18feb0-835d-45e8-a8e2-0053321ce57a-kube-api-access-p899n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.674240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.673969 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.674240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.673987 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.674240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.674005 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.674455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.674436 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.674521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.674438 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.674521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.674467 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.676262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.676242 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f18feb0-835d-45e8-a8e2-0053321ce57a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.676400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.676381 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f18feb0-835d-45e8-a8e2-0053321ce57a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.693625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.693595 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/8f18feb0-835d-45e8-a8e2-0053321ce57a-kube-api-access-p899n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zzjpn\" (UID: \"8f18feb0-835d-45e8-a8e2-0053321ce57a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.853642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.853605 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:23.914243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.914203 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" event={"ID":"d4dfa063-b32f-46d7-975a-378ad1d59be9","Type":"ContainerStarted","Data":"cef77b1538aea941ff486757753eb73e2e807d7b6894d46d73603c8382839f22"} Oct 29 22:26:23.936849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.936758 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:23.959791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:23.959737 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" podStartSLOduration=1.8169218219999999 podStartE2EDuration="2.959718406s" totalImagesPullingTime="1.142796584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:23.957912258 +0000 UTC m=+650.851987178" watchObservedRunningTime="2026-10-29 22:26:23.959718406 +0000 UTC m=+650.853793326" Oct 29 22:26:24.151305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:24.151213 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn"] Oct 29 22:26:24.153803 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:24.153767 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f18feb0_835d_45e8_a8e2_0053321ce57a.slice/crio-98c9b7f64e1d9b08ba6d96e20d84da2a2a985286a06d88506db28bc1a07aef76 WatchSource:0}: Error finding container 98c9b7f64e1d9b08ba6d96e20d84da2a2a985286a06d88506db28bc1a07aef76: Status 404 returned error can't find the container with id 98c9b7f64e1d9b08ba6d96e20d84da2a2a985286a06d88506db28bc1a07aef76 Oct 29 22:26:24.919936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:24.919893 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" event={"ID":"8f18feb0-835d-45e8-a8e2-0053321ce57a","Type":"ContainerStarted","Data":"e34efc2407f9910e7070a9f5900f563abcd40463dfe0c4444ff9542be637f969"} Oct 29 22:26:24.920442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:24.919951 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" event={"ID":"8f18feb0-835d-45e8-a8e2-0053321ce57a","Type":"ContainerStarted","Data":"98c9b7f64e1d9b08ba6d96e20d84da2a2a985286a06d88506db28bc1a07aef76"} Oct 29 22:26:24.966642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:24.966588 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" podStartSLOduration=1.77536299 podStartE2EDuration="1.966573155s" totalImagesPullingTime="191.210165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:24.965210736 +0000 UTC m=+651.859285670" watchObservedRunningTime="2026-10-29 22:26:24.966573155 +0000 UTC m=+651.860648077" Oct 29 22:26:25.853845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:25.853808 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:31.936786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:31.936744 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:31.949816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:31.949773 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:31.962817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:31.962788 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zz4np" Oct 29 22:26:33.854192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:33.854157 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:33.866565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:33.866538 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:33.971894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:33.971864 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" Oct 29 22:26:33.997561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:33.997526 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb"] Oct 29 22:26:34.002614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.002584 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.018430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.018405 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 29 22:26:34.023480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.023450 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb"] Oct 29 22:26:34.089410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.089376 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.089572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.089446 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.089781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.089753 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4c4883-c331-458f-8060-c35e65211fd4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.089842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.089824 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7wh8\" (UniqueName: \"kubernetes.io/projected/ba4c4883-c331-458f-8060-c35e65211fd4-kube-api-access-h7wh8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.089881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.089869 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.089939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.089918 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191041 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191089 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4c4883-c331-458f-8060-c35e65211fd4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191209 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7wh8\" (UniqueName: \"kubernetes.io/projected/ba4c4883-c331-458f-8060-c35e65211fd4-kube-api-access-h7wh8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191253 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191277 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191358 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191695 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191657 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191788 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.191852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.191821 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.193531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.193502 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4c4883-c331-458f-8060-c35e65211fd4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.193671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.193650 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4c4883-c331-458f-8060-c35e65211fd4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.211639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.211612 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7wh8\" (UniqueName: \"kubernetes.io/projected/ba4c4883-c331-458f-8060-c35e65211fd4-kube-api-access-h7wh8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb\" (UID: \"ba4c4883-c331-458f-8060-c35e65211fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.317020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.316979 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:34.609368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.609336 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb"] Oct 29 22:26:34.611963 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:34.611934 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba4c4883_c331_458f_8060_c35e65211fd4.slice/crio-14ec9519649a3c297aacc3be1ce363b0ffdb0d60f1e3cf9d7426bd55afd68492 WatchSource:0}: Error finding container 14ec9519649a3c297aacc3be1ce363b0ffdb0d60f1e3cf9d7426bd55afd68492: Status 404 returned error can't find the container with id 14ec9519649a3c297aacc3be1ce363b0ffdb0d60f1e3cf9d7426bd55afd68492 Oct 29 22:26:34.967335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.967300 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" event={"ID":"ba4c4883-c331-458f-8060-c35e65211fd4","Type":"ContainerStarted","Data":"7079730b795de1c5770abd7aef46d9bbd38d92367d38225aa062fde186d0fd95"} Oct 29 22:26:34.967754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:34.967349 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" event={"ID":"ba4c4883-c331-458f-8060-c35e65211fd4","Type":"ContainerStarted","Data":"14ec9519649a3c297aacc3be1ce363b0ffdb0d60f1e3cf9d7426bd55afd68492"} Oct 29 22:26:35.009589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.009536 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podStartSLOduration=1.8020691 podStartE2EDuration="2.009521116s" totalImagesPullingTime="207.452016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:35.008253592 +0000 UTC m=+661.902328550" watchObservedRunningTime="2026-10-29 22:26:35.009521116 +0000 UTC m=+661.903596037" Oct 29 22:26:35.312423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.312391 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292"] Oct 29 22:26:35.315669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.315651 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.317126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.317102 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:35.331540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.331512 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 29 22:26:35.346159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.346128 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292"] Oct 29 22:26:35.403343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.403302 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.403343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.403349 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.403548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.403372 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m59x2\" (UniqueName: \"kubernetes.io/projected/88e74199-7eba-4437-b7e7-d2dc4094d87e-kube-api-access-m59x2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.403548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.403484 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.403621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.403543 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.403655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.403630 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e74199-7eba-4437-b7e7-d2dc4094d87e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504119 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504166 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504188 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m59x2\" (UniqueName: \"kubernetes.io/projected/88e74199-7eba-4437-b7e7-d2dc4094d87e-kube-api-access-m59x2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504222 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504253 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504274 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e74199-7eba-4437-b7e7-d2dc4094d87e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504606 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504640 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.504800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.504779 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.506734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.506689 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e74199-7eba-4437-b7e7-d2dc4094d87e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.506891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.506875 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e74199-7eba-4437-b7e7-d2dc4094d87e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.524518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.524484 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m59x2\" (UniqueName: \"kubernetes.io/projected/88e74199-7eba-4437-b7e7-d2dc4094d87e-kube-api-access-m59x2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx292\" (UID: \"88e74199-7eba-4437-b7e7-d2dc4094d87e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.629918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.629828 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:35.939619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.939589 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292"] Oct 29 22:26:35.941554 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:35.941532 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e74199_7eba_4437_b7e7_d2dc4094d87e.slice/crio-97e41cf751f87ef536657ba31bca77a2ae4f1d61f2261ae0832cc0ce923ae0fb WatchSource:0}: Error finding container 97e41cf751f87ef536657ba31bca77a2ae4f1d61f2261ae0832cc0ce923ae0fb: Status 404 returned error can't find the container with id 97e41cf751f87ef536657ba31bca77a2ae4f1d61f2261ae0832cc0ce923ae0fb Oct 29 22:26:35.973917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:35.973880 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" event={"ID":"88e74199-7eba-4437-b7e7-d2dc4094d87e","Type":"ContainerStarted","Data":"97e41cf751f87ef536657ba31bca77a2ae4f1d61f2261ae0832cc0ce923ae0fb"} Oct 29 22:26:36.979882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:36.979840 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" event={"ID":"88e74199-7eba-4437-b7e7-d2dc4094d87e","Type":"ContainerStarted","Data":"0975244678b1acb46bd9e2223aef248d3035bcbb0823d5dbcb0b214d58516743"} Oct 29 22:26:37.021248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:37.021187 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" podStartSLOduration=1.8374180070000001 podStartE2EDuration="2.021173953s" totalImagesPullingTime="183.755946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:37.019395387 +0000 UTC m=+663.913470309" watchObservedRunningTime="2026-10-29 22:26:37.021173953 +0000 UTC m=+663.915248906" Oct 29 22:26:37.630328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:37.630285 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:44.317347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:44.317311 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:44.329899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:44.329872 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:45.034043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:45.034015 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:26:45.630118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:45.630084 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:45.642813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:45.642784 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:46.042690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:46.042615 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" Oct 29 22:26:47.389757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.389721 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f97d6577-xkcfk"] Oct 29 22:26:47.392914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.392892 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.412381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.412345 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f97d6577-xkcfk"] Oct 29 22:26:47.520901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.520857 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tgq\" (UniqueName: \"kubernetes.io/projected/611852ef-ca5e-4e8f-b58b-277efda1708c-kube-api-access-j4tgq\") pod \"authorino-7f97d6577-xkcfk\" (UID: \"611852ef-ca5e-4e8f-b58b-277efda1708c\") " pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.520901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.520902 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/611852ef-ca5e-4e8f-b58b-277efda1708c-tls-cert\") pod \"authorino-7f97d6577-xkcfk\" (UID: \"611852ef-ca5e-4e8f-b58b-277efda1708c\") " pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.621765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.621734 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tgq\" (UniqueName: \"kubernetes.io/projected/611852ef-ca5e-4e8f-b58b-277efda1708c-kube-api-access-j4tgq\") pod \"authorino-7f97d6577-xkcfk\" (UID: \"611852ef-ca5e-4e8f-b58b-277efda1708c\") " pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.621765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.621768 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/611852ef-ca5e-4e8f-b58b-277efda1708c-tls-cert\") pod \"authorino-7f97d6577-xkcfk\" (UID: \"611852ef-ca5e-4e8f-b58b-277efda1708c\") " pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.624105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.624078 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/611852ef-ca5e-4e8f-b58b-277efda1708c-tls-cert\") pod \"authorino-7f97d6577-xkcfk\" (UID: \"611852ef-ca5e-4e8f-b58b-277efda1708c\") " pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.641488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.641426 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tgq\" (UniqueName: \"kubernetes.io/projected/611852ef-ca5e-4e8f-b58b-277efda1708c-kube-api-access-j4tgq\") pod \"authorino-7f97d6577-xkcfk\" (UID: \"611852ef-ca5e-4e8f-b58b-277efda1708c\") " pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.704869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.704825 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f97d6577-xkcfk" Oct 29 22:26:47.997256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:47.997200 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f97d6577-xkcfk"] Oct 29 22:26:48.000078 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:48.000046 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod611852ef_ca5e_4e8f_b58b_277efda1708c.slice/crio-b1870afd8440a00f58b2b74280c7b4451ea91b615470f18064d98a8a939be5a4 WatchSource:0}: Error finding container b1870afd8440a00f58b2b74280c7b4451ea91b615470f18064d98a8a939be5a4: Status 404 returned error can't find the container with id b1870afd8440a00f58b2b74280c7b4451ea91b615470f18064d98a8a939be5a4 Oct 29 22:26:48.046963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:48.046925 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f97d6577-xkcfk" event={"ID":"611852ef-ca5e-4e8f-b58b-277efda1708c","Type":"ContainerStarted","Data":"b1870afd8440a00f58b2b74280c7b4451ea91b615470f18064d98a8a939be5a4"} Oct 29 22:26:49.053949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.053919 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f97d6577-xkcfk" event={"ID":"611852ef-ca5e-4e8f-b58b-277efda1708c","Type":"ContainerStarted","Data":"db723188d3877580a96a2cb20750f14e0d05afcc3a494181e83af5d4860615e1"} Oct 29 22:26:49.095477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.095421 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7f97d6577-xkcfk" podStartSLOduration=1.53519603 podStartE2EDuration="2.095405941s" totalImagesPullingTime="560.209911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.094340366 +0000 UTC m=+675.988415288" watchObservedRunningTime="2026-10-29 22:26:49.095405941 +0000 UTC m=+675.989480862" Oct 29 22:26:49.118408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.118369 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cf6f85954-95tgr"] Oct 29 22:26:49.118639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.118606 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5cf6f85954-95tgr" podUID="0bede3ca-7df4-445f-9b88-7cf08843ec2a" containerName="authorino" containerID="cri-o://12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a" gracePeriod=30 Oct 29 22:26:49.393531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.393496 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:26:49.435534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.435494 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert\" (UniqueName: \"kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert\") pod \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " Oct 29 22:26:49.435749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.435613 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr\" (UniqueName: \"kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr\") pod \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\" (UID: \"0bede3ca-7df4-445f-9b88-7cf08843ec2a\") " Oct 29 22:26:49.438174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.438130 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr" pod "0bede3ca-7df4-445f-9b88-7cf08843ec2a" (UID: "0bede3ca-7df4-445f-9b88-7cf08843ec2a"). InnerVolumeSpecName "kube-api-access-2xlrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:49.449807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.449766 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert" pod "0bede3ca-7df4-445f-9b88-7cf08843ec2a" (UID: "0bede3ca-7df4-445f-9b88-7cf08843ec2a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:26:49.537036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.536997 5367 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0bede3ca-7df4-445f-9b88-7cf08843ec2a-tls-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:26:49.537036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:49.537031 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xlrr\" (UniqueName: \"kubernetes.io/projected/0bede3ca-7df4-445f-9b88-7cf08843ec2a-kube-api-access-2xlrr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:26:50.059849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.059810 5367 generic.go:356] "Generic (PLEG): container finished" podID="0bede3ca-7df4-445f-9b88-7cf08843ec2a" containerID="12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a" exitCode=0 Oct 29 22:26:50.060264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.059855 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cf6f85954-95tgr" Oct 29 22:26:50.060264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.059893 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cf6f85954-95tgr" event={"ID":"0bede3ca-7df4-445f-9b88-7cf08843ec2a","Type":"ContainerDied","Data":"12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a"} Oct 29 22:26:50.060264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.059940 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cf6f85954-95tgr" event={"ID":"0bede3ca-7df4-445f-9b88-7cf08843ec2a","Type":"ContainerDied","Data":"e01e4f3be9585e14c843bc486105470fa54e558ad2e7cbcbb9f920df76edc3f3"} Oct 29 22:26:50.060264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.059962 5367 scope.go:122] "RemoveContainer" containerID="12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a" Oct 29 22:26:50.070279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.070253 5367 scope.go:122] "RemoveContainer" containerID="12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a" Oct 29 22:26:50.070596 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:26:50.070577 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a\": container with ID starting with 12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a not found: ID does not exist" containerID="12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a" Oct 29 22:26:50.070665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.070606 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a"} err="failed to get container status \"12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a\": rpc error: code = NotFound desc = could not find container \"12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a\": container with ID starting with 12f0628f10464d5483cf4c1dfc7f92c41b8c3696c69d37329b4e7f01f560810a not found: ID does not exist" Oct 29 22:26:50.118683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.118641 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cf6f85954-95tgr"] Oct 29 22:26:50.122414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:50.122377 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cf6f85954-95tgr"] Oct 29 22:26:51.220657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:51.220623 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bede3ca-7df4-445f-9b88-7cf08843ec2a" path="/var/lib/kubelet/pods/0bede3ca-7df4-445f-9b88-7cf08843ec2a/volumes" Oct 29 22:26:52.334084 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:26:52.334046 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:26:54.916605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.916563 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4"] Oct 29 22:26:54.917102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.917080 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0bede3ca-7df4-445f-9b88-7cf08843ec2a" containerName="authorino" Oct 29 22:26:54.917102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.917100 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bede3ca-7df4-445f-9b88-7cf08843ec2a" containerName="authorino" Oct 29 22:26:54.917233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.917219 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bede3ca-7df4-445f-9b88-7cf08843ec2a" containerName="authorino" Oct 29 22:26:54.921036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.921015 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:54.941538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.941499 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4"] Oct 29 22:26:54.975427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.975401 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 29 22:26:54.976142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.976116 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rb8ck\"" Oct 29 22:26:54.994719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.994669 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqjj\" (UniqueName: \"kubernetes.io/projected/ae46657d-fb4f-4c83-8c75-b38523884064-kube-api-access-kkqjj\") pod \"payload-pre-processing-7b98bb678c-8q5x4\" (UID: \"ae46657d-fb4f-4c83-8c75-b38523884064\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:54.994878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:54.994789 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae46657d-fb4f-4c83-8c75-b38523884064-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8q5x4\" (UID: \"ae46657d-fb4f-4c83-8c75-b38523884064\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:55.095725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:55.095665 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqjj\" (UniqueName: \"kubernetes.io/projected/ae46657d-fb4f-4c83-8c75-b38523884064-kube-api-access-kkqjj\") pod \"payload-pre-processing-7b98bb678c-8q5x4\" (UID: \"ae46657d-fb4f-4c83-8c75-b38523884064\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:55.095898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:55.095751 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae46657d-fb4f-4c83-8c75-b38523884064-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8q5x4\" (UID: \"ae46657d-fb4f-4c83-8c75-b38523884064\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:55.096429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:55.096408 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae46657d-fb4f-4c83-8c75-b38523884064-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8q5x4\" (UID: \"ae46657d-fb4f-4c83-8c75-b38523884064\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:55.115455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:55.115420 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqjj\" (UniqueName: \"kubernetes.io/projected/ae46657d-fb4f-4c83-8c75-b38523884064-kube-api-access-kkqjj\") pod \"payload-pre-processing-7b98bb678c-8q5x4\" (UID: \"ae46657d-fb4f-4c83-8c75-b38523884064\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:55.232945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:55.232862 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:55.531922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:55.531895 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4"] Oct 29 22:26:55.534658 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:55.534629 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae46657d_fb4f_4c83_8c75_b38523884064.slice/crio-0d08ef635a4ab1f906437992376f90cca5594f000ed6604cd67f9eef69dd8fab WatchSource:0}: Error finding container 0d08ef635a4ab1f906437992376f90cca5594f000ed6604cd67f9eef69dd8fab: Status 404 returned error can't find the container with id 0d08ef635a4ab1f906437992376f90cca5594f000ed6604cd67f9eef69dd8fab Oct 29 22:26:56.095117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:56.095077 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" event={"ID":"ae46657d-fb4f-4c83-8c75-b38523884064","Type":"ContainerStarted","Data":"0d08ef635a4ab1f906437992376f90cca5594f000ed6604cd67f9eef69dd8fab"} Oct 29 22:26:57.115534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.115492 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn"] Oct 29 22:26:57.127860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.127827 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.145884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.145792 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn"] Oct 29 22:26:57.158056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.158032 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:26:57.158265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.158247 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lf5dq\"" Oct 29 22:26:57.216329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216289 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216339 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216460 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216498 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216523 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216548 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216571 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216596 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.216693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.216635 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dwr\" (UniqueName: \"kubernetes.io/projected/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-kube-api-access-v7dwr\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318255 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dwr\" (UniqueName: \"kubernetes.io/projected/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-kube-api-access-v7dwr\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318395 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318530 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318629 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318660 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318688 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318745 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318770 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.318880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318796 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.319006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.318942 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.319063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.319013 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.319063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.319051 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.319204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.319185 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.319393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.319373 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.321072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.321047 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.321240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.321221 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.339773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.339734 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.339882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.339849 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dwr\" (UniqueName: \"kubernetes.io/projected/ee2a1d1e-92f2-4ad9-bcff-d180db897e67-kube-api-access-v7dwr\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn\" (UID: \"ee2a1d1e-92f2-4ad9-bcff-d180db897e67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.443691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.443660 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:26:57.751263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:57.751235 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn"] Oct 29 22:26:57.753413 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:26:57.753386 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2a1d1e_92f2_4ad9_bcff_d180db897e67.slice/crio-367159c2428a8d64bd7f7bff4a5aafb4fdba846e7b02af873f58df097baffb72 WatchSource:0}: Error finding container 367159c2428a8d64bd7f7bff4a5aafb4fdba846e7b02af873f58df097baffb72: Status 404 returned error can't find the container with id 367159c2428a8d64bd7f7bff4a5aafb4fdba846e7b02af873f58df097baffb72 Oct 29 22:26:58.106679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:58.106642 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" event={"ID":"ae46657d-fb4f-4c83-8c75-b38523884064","Type":"ContainerStarted","Data":"7334d196a88a876ec5651b4fb2ae1f85062206fc2755737809d9a773230c0c2e"} Oct 29 22:26:58.106905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:58.106773 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:26:58.107954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:58.107931 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" event={"ID":"ee2a1d1e-92f2-4ad9-bcff-d180db897e67","Type":"ContainerStarted","Data":"367159c2428a8d64bd7f7bff4a5aafb4fdba846e7b02af873f58df097baffb72"} Oct 29 22:26:58.147398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:26:58.147334 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" podStartSLOduration=2.3178858350000002 podStartE2EDuration="4.147314609s" totalImagesPullingTime="1.829428774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:58.14595196 +0000 UTC m=+685.040026900" watchObservedRunningTime="2026-10-29 22:26:58.147314609 +0000 UTC m=+685.041389531" Oct 29 22:27:00.207894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:00.207864 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:00.208188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:00.207933 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:00.208188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:00.207961 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:01.125204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:01.125164 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" event={"ID":"ee2a1d1e-92f2-4ad9-bcff-d180db897e67","Type":"ContainerStarted","Data":"b09dfa5eb82313ddc02f669c5ee96f108b89f93ca753c3f78fafb34b31bb366b"} Oct 29 22:27:01.216228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:01.216180 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podStartSLOduration=1.764560317 podStartE2EDuration="4.216167428s" totalImagesPullingTime="2.451607111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:01.202542895 +0000 UTC m=+688.096617815" watchObservedRunningTime="2026-10-29 22:27:01.216167428 +0000 UTC m=+688.110242348" Oct 29 22:27:01.444128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:01.444035 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:27:01.445663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:01.445626 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:01.445807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:01.445687 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:02.444845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:02.444803 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:02.445266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:02.444864 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:03.445017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:03.444975 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:03.445426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:03.445033 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:04.444875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.444835 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:04.445047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.444942 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:04.672723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.661869 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-85bb745f44-xt49q"] Oct 29 22:27:04.682324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.682279 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.685118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.685089 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-85bb745f44-xt49q"] Oct 29 22:27:04.712027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.711955 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 29 22:27:04.712027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.711966 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5vzpj\"" Oct 29 22:27:04.712187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.712027 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 29 22:27:04.804341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.804306 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/87fc931d-3b42-4162-ba49-18e5a06dcb35-kube-api-access-9sfb9\") pod \"ai-gateway-controller-85bb745f44-xt49q\" (UID: \"87fc931d-3b42-4162-ba49-18e5a06dcb35\") " pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.804516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.804478 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc931d-3b42-4162-ba49-18e5a06dcb35-metrics-certs\") pod \"ai-gateway-controller-85bb745f44-xt49q\" (UID: \"87fc931d-3b42-4162-ba49-18e5a06dcb35\") " pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.905917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.905876 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc931d-3b42-4162-ba49-18e5a06dcb35-metrics-certs\") pod \"ai-gateway-controller-85bb745f44-xt49q\" (UID: \"87fc931d-3b42-4162-ba49-18e5a06dcb35\") " pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.905917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.905921 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/87fc931d-3b42-4162-ba49-18e5a06dcb35-kube-api-access-9sfb9\") pod \"ai-gateway-controller-85bb745f44-xt49q\" (UID: \"87fc931d-3b42-4162-ba49-18e5a06dcb35\") " pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.908279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.908252 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc931d-3b42-4162-ba49-18e5a06dcb35-metrics-certs\") pod \"ai-gateway-controller-85bb745f44-xt49q\" (UID: \"87fc931d-3b42-4162-ba49-18e5a06dcb35\") " pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.926223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.926190 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/87fc931d-3b42-4162-ba49-18e5a06dcb35-kube-api-access-9sfb9\") pod \"ai-gateway-controller-85bb745f44-xt49q\" (UID: \"87fc931d-3b42-4162-ba49-18e5a06dcb35\") " pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:04.997413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:04.997325 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:05.298431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:05.298272 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-85bb745f44-xt49q"] Oct 29 22:27:05.300656 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:05.300628 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fc931d_3b42_4162_ba49_18e5a06dcb35.slice/crio-08322d96ec600a09ddc453539034416e6e71ad56a151aebad626d96420594fdc WatchSource:0}: Error finding container 08322d96ec600a09ddc453539034416e6e71ad56a151aebad626d96420594fdc: Status 404 returned error can't find the container with id 08322d96ec600a09ddc453539034416e6e71ad56a151aebad626d96420594fdc Oct 29 22:27:05.444168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:05.444128 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:05.444353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:05.444190 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:06.154863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:06.154821 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" event={"ID":"87fc931d-3b42-4162-ba49-18e5a06dcb35","Type":"ContainerStarted","Data":"08322d96ec600a09ddc453539034416e6e71ad56a151aebad626d96420594fdc"} Oct 29 22:27:06.445095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:06.445009 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:06.445095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:06.445078 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:07.445126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:07.445082 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:27:07.445634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:07.445147 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 29 22:27:08.167228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:08.167187 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" event={"ID":"87fc931d-3b42-4162-ba49-18e5a06dcb35","Type":"ContainerStarted","Data":"5a03b74c3652c20707ac54cb192e3356a029f0a3871a7edb7393d1240e88c9fa"} Oct 29 22:27:08.167432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:08.167291 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:08.209761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:08.209686 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" podStartSLOduration=1.8296306740000001 podStartE2EDuration="4.209667037s" totalImagesPullingTime="2.380036363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:08.206547865 +0000 UTC m=+695.100622786" watchObservedRunningTime="2026-10-29 22:27:08.209667037 +0000 UTC m=+695.103741959" Oct 29 22:27:08.780663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:08.780630 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:27:08.781117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:08.781006 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:27:08.781761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:08.781735 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" Oct 29 22:27:09.117472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:09.117445 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q5x4" Oct 29 22:27:12.156351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.156312 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp"] Oct 29 22:27:12.160123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.160100 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.170739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.170692 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk"] Oct 29 22:27:12.173963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.173945 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.189816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.189789 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp"] Oct 29 22:27:12.218081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.218055 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk"] Oct 29 22:27:12.274611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274577 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.274611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274614 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.274799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274634 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.274799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274723 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.274865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274808 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.274898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274861 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.274898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274882 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.275006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274897 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.275006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274913 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.275006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.275006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.274979 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjg8\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.275132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275052 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.275132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275070 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.275132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275092 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.275219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275135 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcgc\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.275219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275166 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.275219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275193 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.275313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.275283 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.375978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.375938 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.375988 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376009 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376026 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376048 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376110 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376140 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376167 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376213 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376243 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376269 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjg8\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376300 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376324 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376359 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376364 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376397 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcgc\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376423 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376441 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376448 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.376727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376500 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.377143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376858 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.377143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.376881 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.377259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.377157 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.377259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.377165 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.377373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.377311 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.377442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.377419 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.377909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.377882 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.378639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.378614 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.379080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.379056 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.379171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.379155 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.379281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.379263 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.396128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.396098 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.396248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.396195 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcgc\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.396350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.396326 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.396441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.396426 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjg8\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-45vtp\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.473618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.473523 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:12.491863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.491835 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:12.798163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.798135 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp"] Oct 29 22:27:12.800370 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.800342 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk"] Oct 29 22:27:12.801116 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:12.801092 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0a4407_0222_426a_bc2e_49c63ac7f58a.slice/crio-08a9bd2c783a3614aca74eaf0429de2648e77b9ae4ebb70adf17ffe6aa8294f2 WatchSource:0}: Error finding container 08a9bd2c783a3614aca74eaf0429de2648e77b9ae4ebb70adf17ffe6aa8294f2: Status 404 returned error can't find the container with id 08a9bd2c783a3614aca74eaf0429de2648e77b9ae4ebb70adf17ffe6aa8294f2 Oct 29 22:27:12.802488 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:12.802457 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950c8f41_a5b6_483a_a31b_2edc57387dfe.slice/crio-6aa01bb1595f52828e656a18726334c62d0af452f683bc5e359f4398008ffbf6 WatchSource:0}: Error finding container 6aa01bb1595f52828e656a18726334c62d0af452f683bc5e359f4398008ffbf6: Status 404 returned error can't find the container with id 6aa01bb1595f52828e656a18726334c62d0af452f683bc5e359f4398008ffbf6 Oct 29 22:27:12.803325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.803289 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:12.803423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.803376 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:12.803472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.803418 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:12.804617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.804589 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:12.804732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.804656 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:12.804791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:12.804725 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:13.197772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.197737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" event={"ID":"950c8f41-a5b6-483a-a31b-2edc57387dfe","Type":"ContainerStarted","Data":"3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21"} Oct 29 22:27:13.198196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.197798 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" event={"ID":"950c8f41-a5b6-483a-a31b-2edc57387dfe","Type":"ContainerStarted","Data":"6aa01bb1595f52828e656a18726334c62d0af452f683bc5e359f4398008ffbf6"} Oct 29 22:27:13.199778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.199751 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" event={"ID":"be0a4407-0222-426a-bc2e-49c63ac7f58a","Type":"ContainerStarted","Data":"57122ff1d1a88e44b21922d7ce44504d0748b90c60eb02b07842f7ab58e19793"} Oct 29 22:27:13.199892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.199795 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" event={"ID":"be0a4407-0222-426a-bc2e-49c63ac7f58a","Type":"ContainerStarted","Data":"08a9bd2c783a3614aca74eaf0429de2648e77b9ae4ebb70adf17ffe6aa8294f2"} Oct 29 22:27:13.249259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.249189 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podStartSLOduration=1.249171134 podStartE2EDuration="1.249171134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:13.246557302 +0000 UTC m=+700.140632299" watchObservedRunningTime="2026-10-29 22:27:13.249171134 +0000 UTC m=+700.143246056" Oct 29 22:27:13.320178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.320128 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podStartSLOduration=1.320114489 podStartE2EDuration="1.320114489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:13.318420395 +0000 UTC m=+700.212495307" watchObservedRunningTime="2026-10-29 22:27:13.320114489 +0000 UTC m=+700.214189410" Oct 29 22:27:13.476667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.476577 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:13.477026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.476997 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:13.477204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.477178 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:13.496270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.496235 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:13.496846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.496803 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:13.497001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:13.496868 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:14.482723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:14.482093 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:14.482723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:14.482155 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:14.495727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:14.493082 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:14.495727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:14.493146 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:15.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:15.475144 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:15.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:15.475228 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:15.496910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:15.496873 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:15.497498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:15.497469 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:16.474723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:16.474114 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:16.474723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:16.474192 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:16.496594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:16.496560 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:16.496876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:16.496852 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:17.476723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:17.476102 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:17.476723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:17.476164 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:17.497724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:17.495615 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:17.497724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:17.495675 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:18.475721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:18.475099 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:18.475721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:18.475162 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:18.497721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:18.497588 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:18.497721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:18.497645 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:19.174883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:19.174851 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-85bb745f44-xt49q" Oct 29 22:27:19.475153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:19.475066 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:19.475153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:19.475137 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 29 22:27:19.492526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:19.492489 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 29 22:27:19.492608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:19.492566 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 29 22:27:21.161063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:21.161023 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:21.162199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:21.162173 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:21.162761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:21.162743 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 22:27:21.181437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:21.181416 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:21.181838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:21.181817 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:21.182542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:21.182526 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:27:22.330261 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:27:22.330221 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:27:47.414482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.414409 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789"] Oct 29 22:27:47.417979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.417960 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.434528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.434507 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8d3632-openshift-default-dockercfg-x5w92\"" Oct 29 22:27:47.450829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.450800 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789"] Oct 29 22:27:47.529352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529319 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529356 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529379 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529402 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529564 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgp55\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529626 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529655 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529693 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.529926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.529752 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.568200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.568170 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk"] Oct 29 22:27:47.572162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.572140 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.588172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.588150 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4a0287-openshift-default-dockercfg-dcw9p\"" Oct 29 22:27:47.600196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.600175 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk"] Oct 29 22:27:47.630438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630411 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630443 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630484 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630510 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630621 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630654 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630687 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630740 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630793 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgp55\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630881 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.630976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630955 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.631102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.630985 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.631289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.631266 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.631574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.631523 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.632709 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.632682 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.633492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.633466 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.653817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.653793 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgp55\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.654406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.654381 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.731958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.731869 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5dj\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.731958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.731908 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.731958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.731931 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.732173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.732002 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.732173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.732081 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.732173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.732145 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.732267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.732187 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.732267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.732239 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.732326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.732285 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.735138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.735105 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:47.832800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.832770 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.833028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.832814 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.833028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.832873 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.833028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.832958 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5dj\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.833028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.832988 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.833028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.833025 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.834081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.833050 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.834081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.833078 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.834081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.833111 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.835637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.835609 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.836034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.836009 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.836212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.836180 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.836347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.836328 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.836446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.836424 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.836915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.836891 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.837180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.837159 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28"] Oct 29 22:27:47.837797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.837776 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.842087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.842063 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.863497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.863475 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.868114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.868078 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28"] Oct 29 22:27:47.868302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.868286 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-92fd86-openshift-default-dockercfg-485tp\"" Oct 29 22:27:47.870590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.870566 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5dj\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.885924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.885885 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:47.934901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.934875 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m86c\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.934942 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935028 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935059 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935123 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935168 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935322 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935385 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:47.935455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:47.935430 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036279 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036320 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036368 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036399 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036438 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036536 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036579 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036629 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036659 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m86c\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.036910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.036875 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.037364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.037339 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.037807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.037786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.037914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.037781 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.037914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.037818 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.038953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.038933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.039052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.038972 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.053980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.053962 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789"] Oct 29 22:27:48.056210 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:48.056182 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86451d4b_a607_4d8e_90ef_ab2267aa7705.slice/crio-70f2d991c263aac9e13f013e881a2e139f65209dbe37f2a75d96ffb7d0931f32 WatchSource:0}: Error finding container 70f2d991c263aac9e13f013e881a2e139f65209dbe37f2a75d96ffb7d0931f32: Status 404 returned error can't find the container with id 70f2d991c263aac9e13f013e881a2e139f65209dbe37f2a75d96ffb7d0931f32 Oct 29 22:27:48.057909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.057818 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.058111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.058090 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m86c\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.058624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.058591 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.058748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.058670 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.058819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.058741 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.155962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.155932 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:48.164504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.164471 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk"] Oct 29 22:27:48.168576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.168551 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.187404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.187371 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk"] Oct 29 22:27:48.188008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.187991 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0d12c5-openshift-default-dockercfg-477sq\"" Oct 29 22:27:48.195277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.195238 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk"] Oct 29 22:27:48.196110 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:48.196068 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7fed1d_bb2f_44e3_b1af_6043e5208157.slice/crio-60fc0f745f1f920a37c453f16006996e4129708a4c813c01f0b286ac86b96b79 WatchSource:0}: Error finding container 60fc0f745f1f920a37c453f16006996e4129708a4c813c01f0b286ac86b96b79: Status 404 returned error can't find the container with id 60fc0f745f1f920a37c453f16006996e4129708a4c813c01f0b286ac86b96b79 Oct 29 22:27:48.199195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.199154 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.199311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.199215 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.199311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.199247 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.238683 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.239913 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.239947 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.239975 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.240079 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.240139 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.240192 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.240216 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.242403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.240242 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vrc\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341517 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341568 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341601 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341677 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341746 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341790 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341816 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341840 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vrc\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.341977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.341961 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.342462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.342437 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.343402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.343377 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.344010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.343983 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.344606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.344580 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.345535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.345510 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.346715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.346679 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.348147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.348125 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.366016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.365990 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.366455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.366431 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vrc\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.376162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.376135 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" event={"ID":"3b7fed1d-bb2f-44e3-b1af-6043e5208157","Type":"ContainerStarted","Data":"e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11"} Oct 29 22:27:48.376274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.376190 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" event={"ID":"3b7fed1d-bb2f-44e3-b1af-6043e5208157","Type":"ContainerStarted","Data":"60fc0f745f1f920a37c453f16006996e4129708a4c813c01f0b286ac86b96b79"} Oct 29 22:27:48.377883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.377860 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" event={"ID":"86451d4b-a607-4d8e-90ef-ab2267aa7705","Type":"ContainerStarted","Data":"19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234"} Oct 29 22:27:48.377984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.377899 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" event={"ID":"86451d4b-a607-4d8e-90ef-ab2267aa7705","Type":"ContainerStarted","Data":"70f2d991c263aac9e13f013e881a2e139f65209dbe37f2a75d96ffb7d0931f32"} Oct 29 22:27:48.422722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.422647 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podStartSLOduration=1.422629323 podStartE2EDuration="1.422629323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:48.421244447 +0000 UTC m=+735.315319369" watchObservedRunningTime="2026-10-29 22:27:48.422629323 +0000 UTC m=+735.316704247" Oct 29 22:27:48.479633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.479559 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podStartSLOduration=1.479541528 podStartE2EDuration="1.479541528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:48.470082675 +0000 UTC m=+735.364157598" watchObservedRunningTime="2026-10-29 22:27:48.479541528 +0000 UTC m=+735.373616450" Oct 29 22:27:48.480747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.480721 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28"] Oct 29 22:27:48.480922 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:48.480896 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff1f6bc_160e_4ab6_815d_c344cd149a65.slice/crio-7b0da99035abadd77bea8a3b0f0530bd1e91ea55936fd561cb451fd204ebb1f1 WatchSource:0}: Error finding container 7b0da99035abadd77bea8a3b0f0530bd1e91ea55936fd561cb451fd204ebb1f1: Status 404 returned error can't find the container with id 7b0da99035abadd77bea8a3b0f0530bd1e91ea55936fd561cb451fd204ebb1f1 Oct 29 22:27:48.483231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.483197 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.483314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.483276 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.483351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.483321 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.496452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.496426 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:48.735679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.735606 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:48.797168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.797147 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk"] Oct 29 22:27:48.799349 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:48.799312 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe5c579_0c5f_4404_be47_1c61db827670.slice/crio-15f5df4faafd6deff9b354fff6ad434389b1692fe44d4aec5030ecf8bfbb6988 WatchSource:0}: Error finding container 15f5df4faafd6deff9b354fff6ad434389b1692fe44d4aec5030ecf8bfbb6988: Status 404 returned error can't find the container with id 15f5df4faafd6deff9b354fff6ad434389b1692fe44d4aec5030ecf8bfbb6988 Oct 29 22:27:48.801647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.801613 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.801747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.801680 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.801747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.801727 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:27:48.886233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:48.886214 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:49.384267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.384233 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" event={"ID":"0fe5c579-0c5f-4404-be47-1c61db827670","Type":"ContainerStarted","Data":"4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f"} Oct 29 22:27:49.384451 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.384289 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" event={"ID":"0fe5c579-0c5f-4404-be47-1c61db827670","Type":"ContainerStarted","Data":"15f5df4faafd6deff9b354fff6ad434389b1692fe44d4aec5030ecf8bfbb6988"} Oct 29 22:27:49.385770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.385742 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" event={"ID":"aff1f6bc-160e-4ab6-815d-c344cd149a65","Type":"ContainerStarted","Data":"bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648"} Oct 29 22:27:49.385871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.385788 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" event={"ID":"aff1f6bc-160e-4ab6-815d-c344cd149a65","Type":"ContainerStarted","Data":"7b0da99035abadd77bea8a3b0f0530bd1e91ea55936fd561cb451fd204ebb1f1"} Oct 29 22:27:49.428043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.427996 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podStartSLOduration=1.427982788 podStartE2EDuration="1.427982788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:49.426754958 +0000 UTC m=+736.320829880" watchObservedRunningTime="2026-10-29 22:27:49.427982788 +0000 UTC m=+736.322057709" Oct 29 22:27:49.473593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.473546 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podStartSLOduration=2.473531341 podStartE2EDuration="2.473531341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:49.471348306 +0000 UTC m=+736.365423228" watchObservedRunningTime="2026-10-29 22:27:49.473531341 +0000 UTC m=+736.367606258" Oct 29 22:27:49.497536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.497501 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:49.502523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.502498 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:49.741547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.741467 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:49.892309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:49.892277 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:50.157130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.157098 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:50.162079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.162053 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:50.391795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.391761 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:50.392109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.391824 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:50.392504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.392298 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:50.392504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.392339 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:50.392883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.392862 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:27:50.392977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.392925 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:27:50.392977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.392972 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:27:50.393054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:50.393046 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:27:52.330038 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:27:52.330004 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:27:52.715683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.715605 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9"] Oct 29 22:27:52.719368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.719350 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:52.742451 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.742416 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9"] Oct 29 22:27:52.751932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.751901 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4a0287\"" Oct 29 22:27:52.751932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.751912 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4a0287-dockercfg-m6hjg\"" Oct 29 22:27:52.794036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.794005 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:52.794193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.794055 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfqt\" (UniqueName: \"kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt\") pod \"payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:52.894914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.894876 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfqt\" (UniqueName: \"kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt\") pod \"payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:52.895112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.895079 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:52.895876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.895847 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:52.915293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:52.915268 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfqt\" (UniqueName: \"kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt\") pod \"payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:53.033441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:53.033369 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:53.335946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:53.335916 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9"] Oct 29 22:27:53.411253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:53.411225 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" event={"ID":"c8558681-9856-4931-a120-1008a32b6300","Type":"ContainerStarted","Data":"0cb1affb54dba8da45eea62b304d05b18bc9c2b85839ed5d76e688a220937db1"} Oct 29 22:27:53.411383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:53.411365 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:27:53.471888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:53.471834 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" podStartSLOduration=1.47181976 podStartE2EDuration="1.47181976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:53.469744389 +0000 UTC m=+740.363819306" watchObservedRunningTime="2026-10-29 22:27:53.47181976 +0000 UTC m=+740.365894681" Oct 29 22:27:54.403031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.402997 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s"] Oct 29 22:27:54.407721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.407555 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.418122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.418090 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" event={"ID":"c8558681-9856-4931-a120-1008a32b6300","Type":"ContainerStarted","Data":"b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e"} Oct 29 22:27:54.426724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.426680 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s"] Oct 29 22:27:54.437240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.437210 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8d3632\"" Oct 29 22:27:54.479112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.479081 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8d3632-dockercfg-7mp7c\"" Oct 29 22:27:54.510555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.510516 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.510751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.510602 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd65h\" (UniqueName: \"kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h\") pod \"payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.612170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.612129 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.612315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.612224 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd65h\" (UniqueName: \"kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h\") pod \"payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.612882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.612853 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.635259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.635234 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd65h\" (UniqueName: \"kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h\") pod \"payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:54.722618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:54.722541 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:55.016771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.016743 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s"] Oct 29 22:27:55.018992 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:55.018962 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eae67af_3766_43aa_a36e_b088f5479e2f.slice/crio-0552cef14841380deaf5981edb29fc2a88f615eba95ac7d1a82297d9bd8af5b5 WatchSource:0}: Error finding container 0552cef14841380deaf5981edb29fc2a88f615eba95ac7d1a82297d9bd8af5b5: Status 404 returned error can't find the container with id 0552cef14841380deaf5981edb29fc2a88f615eba95ac7d1a82297d9bd8af5b5 Oct 29 22:27:55.243340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.243307 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp"] Oct 29 22:27:55.246917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.246893 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.268084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.268028 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp"] Oct 29 22:27:55.318371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.318345 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-40cf9a\"" Oct 29 22:27:55.318653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.318637 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-40cf9a-dockercfg-vjgvl\"" Oct 29 22:27:55.318729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.318638 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.318729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.318684 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljwg\" (UniqueName: \"kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg\") pod \"payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.429435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.429313 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.430197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.430172 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljwg\" (UniqueName: \"kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg\") pod \"payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.436994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.432527 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.440323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.440295 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" event={"ID":"6eae67af-3766-43aa-a36e-b088f5479e2f","Type":"ContainerStarted","Data":"5690883cd09be426793c228bb22c289dc660df6e26155fcba22de6a7f146ecd6"} Oct 29 22:27:55.440459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.440341 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" event={"ID":"6eae67af-3766-43aa-a36e-b088f5479e2f","Type":"ContainerStarted","Data":"0552cef14841380deaf5981edb29fc2a88f615eba95ac7d1a82297d9bd8af5b5"} Oct 29 22:27:55.440459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.440377 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:27:55.450999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.450975 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljwg\" (UniqueName: \"kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg\") pod \"payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.480271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.480212 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" podStartSLOduration=1.480194528 podStartE2EDuration="1.480194528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:55.479190095 +0000 UTC m=+742.373265017" watchObservedRunningTime="2026-10-29 22:27:55.480194528 +0000 UTC m=+742.374269451" Oct 29 22:27:55.559933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.559910 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:55.863484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:55.863456 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp"] Oct 29 22:27:55.865284 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:55.865260 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode039e8c1_7cae_4480_97bd_4981a3425750.slice/crio-a3e407541a68fbcf5705b6c61d4708defb0cacd06ae8dbe4ccec9bcc1ef23a4d WatchSource:0}: Error finding container a3e407541a68fbcf5705b6c61d4708defb0cacd06ae8dbe4ccec9bcc1ef23a4d: Status 404 returned error can't find the container with id a3e407541a68fbcf5705b6c61d4708defb0cacd06ae8dbe4ccec9bcc1ef23a4d Oct 29 22:27:56.448533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.448489 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" event={"ID":"e039e8c1-7cae-4480-97bd-4981a3425750","Type":"ContainerStarted","Data":"56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7"} Oct 29 22:27:56.448995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.448549 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" event={"ID":"e039e8c1-7cae-4480-97bd-4981a3425750","Type":"ContainerStarted","Data":"a3e407541a68fbcf5705b6c61d4708defb0cacd06ae8dbe4ccec9bcc1ef23a4d"} Oct 29 22:27:56.448995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.448730 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:27:56.488356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.488302 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" podStartSLOduration=1.488286573 podStartE2EDuration="1.488286573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:56.487442522 +0000 UTC m=+743.381517445" watchObservedRunningTime="2026-10-29 22:27:56.488286573 +0000 UTC m=+743.382361495" Oct 29 22:27:56.694413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.694375 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d"] Oct 29 22:27:56.697824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.697807 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.713766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.713741 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-0d12c5-dockercfg-p7b2j\"" Oct 29 22:27:56.719597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.719569 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0d12c5\"" Oct 29 22:27:56.720234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.720212 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d"] Oct 29 22:27:56.743432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.743405 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb"] Oct 29 22:27:56.746886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.746869 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:56.767759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.753743 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.767759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.753925 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p94c4\" (UniqueName: \"kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4\") pod \"payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.788260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.788228 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb"] Oct 29 22:27:56.876725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.863269 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.876725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.863433 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p94c4\" (UniqueName: \"kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4\") pod \"payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.876725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.863469 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:56.876725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.863525 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gh9t\" (UniqueName: \"kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t\") pod \"payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:56.876725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.864499 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.884630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.884600 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94c4\" (UniqueName: \"kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4\") pod \"payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:56.978773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.975799 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:56.978773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.975878 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gh9t\" (UniqueName: \"kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t\") pod \"payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:56.978773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.977171 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:56.996917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:56.996891 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gh9t\" (UniqueName: \"kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t\") pod \"payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:57.014108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.010967 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:57.094539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.094508 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:57.330161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.330132 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d"] Oct 29 22:27:57.331759 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:57.331734 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52d37f9_e67b_43a3_849f_0b9f0077f13d.slice/crio-e14a9a4d1fe2514b4009ef9a5586d90312b9e707b51dc0248dda08f08e4e8dfe WatchSource:0}: Error finding container e14a9a4d1fe2514b4009ef9a5586d90312b9e707b51dc0248dda08f08e4e8dfe: Status 404 returned error can't find the container with id e14a9a4d1fe2514b4009ef9a5586d90312b9e707b51dc0248dda08f08e4e8dfe Oct 29 22:27:57.422815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.422788 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb"] Oct 29 22:27:57.424047 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:57.424021 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd57bb9_a797_4434_be8a_4550badfdbeb.slice/crio-31c6c4e9165de629619ca275fe0c8cc588a78f2d7fd285525b37eb13bbd327c2 WatchSource:0}: Error finding container 31c6c4e9165de629619ca275fe0c8cc588a78f2d7fd285525b37eb13bbd327c2: Status 404 returned error can't find the container with id 31c6c4e9165de629619ca275fe0c8cc588a78f2d7fd285525b37eb13bbd327c2 Oct 29 22:27:57.482378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.482324 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" event={"ID":"afd57bb9-a797-4434-be8a-4550badfdbeb","Type":"ContainerStarted","Data":"31c6c4e9165de629619ca275fe0c8cc588a78f2d7fd285525b37eb13bbd327c2"} Oct 29 22:27:57.486481 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.485897 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" event={"ID":"e52d37f9-e67b-43a3-849f-0b9f0077f13d","Type":"ContainerStarted","Data":"02511be994abc0527f89cc533f1acb390d70c835e8c8365c045b46dea95c469e"} Oct 29 22:27:57.486481 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.485986 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" event={"ID":"e52d37f9-e67b-43a3-849f-0b9f0077f13d","Type":"ContainerStarted","Data":"e14a9a4d1fe2514b4009ef9a5586d90312b9e707b51dc0248dda08f08e4e8dfe"} Oct 29 22:27:57.486824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.486803 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:27:57.528919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:57.528875 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" podStartSLOduration=1.5288621550000001 podStartE2EDuration="1.528862155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:57.527901933 +0000 UTC m=+744.421976857" watchObservedRunningTime="2026-10-29 22:27:57.528862155 +0000 UTC m=+744.422937073" Oct 29 22:27:58.315052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.315014 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz"] Oct 29 22:27:58.318720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.318682 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.338217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.338183 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz"] Oct 29 22:27:58.338850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.338830 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-92fd86-dockercfg-qsgt8\"" Oct 29 22:27:58.402285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.402245 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.402468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.402348 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8thg\" (UniqueName: \"kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg\") pod \"payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.420869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.420834 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8"] Oct 29 22:27:58.421919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.421895 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-92fd86\"" Oct 29 22:27:58.426144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.426118 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.443776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.443744 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8"] Oct 29 22:27:58.493771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.493735 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" event={"ID":"afd57bb9-a797-4434-be8a-4550badfdbeb","Type":"ContainerStarted","Data":"cad417c5cc294f2fe00ba19e6d530f815fd84fcda89aa756d1983dc7ef379fb3"} Oct 29 22:27:58.494358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.493952 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:27:58.504073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.504043 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.504189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.504115 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8thg\" (UniqueName: \"kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg\") pod \"payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.504799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.504781 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.524134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.524108 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8thg\" (UniqueName: \"kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg\") pod \"payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.549304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.549262 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" podStartSLOduration=2.549249966 podStartE2EDuration="2.549249966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:58.547873445 +0000 UTC m=+745.441948367" watchObservedRunningTime="2026-10-29 22:27:58.549249966 +0000 UTC m=+745.443324926" Oct 29 22:27:58.605636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.605559 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q74x\" (UniqueName: \"kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x\") pod \"payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.605780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.605687 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.632951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.632928 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:58.708847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.707674 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q74x\" (UniqueName: \"kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x\") pod \"payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.708847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.708096 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.709092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.709043 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.732826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.732764 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q74x\" (UniqueName: \"kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x\") pod \"payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.742685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.742550 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:58.984220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:58.984080 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz"] Oct 29 22:27:58.990303 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:58.990271 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90656a9d_0221_4d3d_b244_1a0d1ff321a6.slice/crio-7a951b8d4ea88ef936266e04597b620a13df144ba97b6e5d5d6592cc1293fea6 WatchSource:0}: Error finding container 7a951b8d4ea88ef936266e04597b620a13df144ba97b6e5d5d6592cc1293fea6: Status 404 returned error can't find the container with id 7a951b8d4ea88ef936266e04597b620a13df144ba97b6e5d5d6592cc1293fea6 Oct 29 22:27:59.083156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.083081 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8"] Oct 29 22:27:59.084844 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:59.084813 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ab836b_2090_498b_99b5_34f1256aa9b1.slice/crio-4c09ec888953f05dff1dbb05afa164de07760b013062096064622792b14060d5 WatchSource:0}: Error finding container 4c09ec888953f05dff1dbb05afa164de07760b013062096064622792b14060d5: Status 404 returned error can't find the container with id 4c09ec888953f05dff1dbb05afa164de07760b013062096064622792b14060d5 Oct 29 22:27:59.085364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.085345 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb"] Oct 29 22:27:59.089486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.089469 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.105554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.105528 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb"] Oct 29 22:27:59.106427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.106408 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-95672c\"" Oct 29 22:27:59.106519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.106508 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-95672c-dockercfg-wzlqz\"" Oct 29 22:27:59.150221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.150197 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx"] Oct 29 22:27:59.182721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.170096 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.185737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.185693 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx"] Oct 29 22:27:59.225724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.224042 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx85w\" (UniqueName: \"kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w\") pod \"payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.225724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.224161 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.324868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.324830 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.325068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.324946 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnpb\" (UniqueName: \"kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb\") pod \"payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.325068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.325054 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx85w\" (UniqueName: \"kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w\") pod \"payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.325153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.325124 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.325586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.325565 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.346616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.346589 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx85w\" (UniqueName: \"kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w\") pod \"payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.425983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.425948 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.426134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.426033 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnpb\" (UniqueName: \"kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb\") pod \"payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.426784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.426762 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.440984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.440954 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:27:59.446902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.446871 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnpb\" (UniqueName: \"kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb\") pod \"payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.498250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.498222 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:27:59.516118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.516080 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" event={"ID":"90656a9d-0221-4d3d-b244-1a0d1ff321a6","Type":"ContainerStarted","Data":"bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24"} Oct 29 22:27:59.516197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.516185 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" event={"ID":"90656a9d-0221-4d3d-b244-1a0d1ff321a6","Type":"ContainerStarted","Data":"7a951b8d4ea88ef936266e04597b620a13df144ba97b6e5d5d6592cc1293fea6"} Oct 29 22:27:59.516362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.516344 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:27:59.519082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.519055 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" event={"ID":"44ab836b-2090-498b-99b5-34f1256aa9b1","Type":"ContainerStarted","Data":"58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a"} Oct 29 22:27:59.519172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.519107 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:27:59.519172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.519130 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" event={"ID":"44ab836b-2090-498b-99b5-34f1256aa9b1","Type":"ContainerStarted","Data":"4c09ec888953f05dff1dbb05afa164de07760b013062096064622792b14060d5"} Oct 29 22:27:59.570733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.568520 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" podStartSLOduration=1.5685030819999999 podStartE2EDuration="1.568503082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:59.567876215 +0000 UTC m=+746.461951138" watchObservedRunningTime="2026-10-29 22:27:59.568503082 +0000 UTC m=+746.462578006" Oct 29 22:27:59.786966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.785816 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" podStartSLOduration=1.785796515 podStartE2EDuration="1.785796515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:59.615242368 +0000 UTC m=+746.509317289" watchObservedRunningTime="2026-10-29 22:27:59.785796515 +0000 UTC m=+746.679871437" Oct 29 22:27:59.786966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.786230 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb"] Oct 29 22:27:59.886036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.884153 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx"] Oct 29 22:27:59.890437 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:27:59.890411 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b776b7_051e_4b07_8c62_c76f2c210ad9.slice/crio-14c1557ec1541a0b2c840ebecf867f85399831b92b38cc870280d1f0653b5fb4 WatchSource:0}: Error finding container 14c1557ec1541a0b2c840ebecf867f85399831b92b38cc870280d1f0653b5fb4: Status 404 returned error can't find the container with id 14c1557ec1541a0b2c840ebecf867f85399831b92b38cc870280d1f0653b5fb4 Oct 29 22:27:59.987177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.987146 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj"] Oct 29 22:27:59.991484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:27:59.991458 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.004470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.004441 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj"] Oct 29 22:28:00.007476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.007451 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-879596\"" Oct 29 22:28:00.007568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.007548 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-879596-dockercfg-kv67z\"" Oct 29 22:28:00.082722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.082179 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf"] Oct 29 22:28:00.129736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.128885 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf"] Oct 29 22:28:00.129736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.129022 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.135198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.135162 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.135343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.135212 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wng9\" (UniqueName: \"kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9\") pod \"payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.236437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.236402 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqmj\" (UniqueName: \"kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj\") pod \"payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.236632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.236456 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.236632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.236498 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.236632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.236535 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wng9\" (UniqueName: \"kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9\") pod \"payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.237527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.237500 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.278758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.271343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wng9\" (UniqueName: \"kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9\") pod \"payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.308403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.308367 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:00.338647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.337344 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.338647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.337624 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqmj\" (UniqueName: \"kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj\") pod \"payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.338647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.338216 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.381476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.381405 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqmj\" (UniqueName: \"kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj\") pod \"payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.446471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.446431 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:00.530982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.529253 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" event={"ID":"c0b776b7-051e-4b07-8c62-c76f2c210ad9","Type":"ContainerStarted","Data":"8fee0a5e0ce8233c630bdd36268557cf76fd27bf38e138914f311cf673227fea"} Oct 29 22:28:00.530982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.529316 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" event={"ID":"c0b776b7-051e-4b07-8c62-c76f2c210ad9","Type":"ContainerStarted","Data":"14c1557ec1541a0b2c840ebecf867f85399831b92b38cc870280d1f0653b5fb4"} Oct 29 22:28:00.530982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.530780 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:28:00.535002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.534973 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" event={"ID":"acffe8f2-1085-40f6-9093-f916144f64c1","Type":"ContainerStarted","Data":"a80944571a614d7a5bcdfeb2fdbc2aaab52e4a43fe29586c8fee68580533932d"} Oct 29 22:28:00.536023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.536002 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" event={"ID":"acffe8f2-1085-40f6-9093-f916144f64c1","Type":"ContainerStarted","Data":"05abbf855d24e7ee73263faa8a32637ddd8570a4fe524104cd39ad305a6484b9"} Oct 29 22:28:00.536260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.536244 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:28:00.584725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.583579 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" podStartSLOduration=1.583558151 podStartE2EDuration="1.583558151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:00.580140811 +0000 UTC m=+747.474215734" watchObservedRunningTime="2026-10-29 22:28:00.583558151 +0000 UTC m=+747.477633074" Oct 29 22:28:00.622648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.622598 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" podStartSLOduration=1.622585137 podStartE2EDuration="1.622585137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:00.620652696 +0000 UTC m=+747.514727609" watchObservedRunningTime="2026-10-29 22:28:00.622585137 +0000 UTC m=+747.516660057" Oct 29 22:28:00.642642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.642581 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj"] Oct 29 22:28:00.642801 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:28:00.642756 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0b05f3_9543_4a70_ac49_175571ea9adf.slice/crio-072f3897c2c412212968cf70f00cc7358293d6e533ee830dfc740e683df72b7e WatchSource:0}: Error finding container 072f3897c2c412212968cf70f00cc7358293d6e533ee830dfc740e683df72b7e: Status 404 returned error can't find the container with id 072f3897c2c412212968cf70f00cc7358293d6e533ee830dfc740e683df72b7e Oct 29 22:28:00.782680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:00.782260 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf"] Oct 29 22:28:00.784295 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:28:00.784262 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb474646f_cd1b_4689_a935_46b6468ab79c.slice/crio-cc9f4311ed8748537bb27659d22a1dfcf0055bb080152c17eed34292f91b09c4 WatchSource:0}: Error finding container cc9f4311ed8748537bb27659d22a1dfcf0055bb080152c17eed34292f91b09c4: Status 404 returned error can't find the container with id cc9f4311ed8748537bb27659d22a1dfcf0055bb080152c17eed34292f91b09c4 Oct 29 22:28:01.541285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.541247 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" event={"ID":"cd0b05f3-9543-4a70-ac49-175571ea9adf","Type":"ContainerStarted","Data":"a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f"} Oct 29 22:28:01.541759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.541301 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" event={"ID":"cd0b05f3-9543-4a70-ac49-175571ea9adf","Type":"ContainerStarted","Data":"072f3897c2c412212968cf70f00cc7358293d6e533ee830dfc740e683df72b7e"} Oct 29 22:28:01.541759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.541353 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:01.542985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.542962 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" event={"ID":"b474646f-cd1b-4689-a935-46b6468ab79c","Type":"ContainerStarted","Data":"8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1"} Oct 29 22:28:01.543072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.543002 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" event={"ID":"b474646f-cd1b-4689-a935-46b6468ab79c","Type":"ContainerStarted","Data":"cc9f4311ed8748537bb27659d22a1dfcf0055bb080152c17eed34292f91b09c4"} Oct 29 22:28:01.543142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.543131 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:01.601036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.600979 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" podStartSLOduration=2.600963622 podStartE2EDuration="2.600963622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:01.599263053 +0000 UTC m=+748.493337972" watchObservedRunningTime="2026-10-29 22:28:01.600963622 +0000 UTC m=+748.495038563" Oct 29 22:28:01.646002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:01.645950 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" podStartSLOduration=1.645933018 podStartE2EDuration="1.645933018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:01.643784321 +0000 UTC m=+748.537859243" watchObservedRunningTime="2026-10-29 22:28:01.645933018 +0000 UTC m=+748.540007939" Oct 29 22:28:05.442131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.442100 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 22:28:05.531670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.531636 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:33662->10.133.0.48:15021: read: connection reset by peer" start-of-body= Oct 29 22:28:05.531925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.531888 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:33662->10.133.0.48:15021: read: connection reset by peer" Oct 29 22:28:05.575856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.575811 5367 generic.go:356] "Generic (PLEG): container finished" podID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerID="19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234" exitCode=0 Oct 29 22:28:05.576016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.575872 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" event={"ID":"86451d4b-a607-4d8e-90ef-ab2267aa7705","Type":"ContainerDied","Data":"19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234"} Oct 29 22:28:05.576362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.576344 5367 scope.go:122] "RemoveContainer" containerID="19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234" Oct 29 22:28:05.578072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.578025 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:05.578189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.578141 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:05.578257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:05.578197 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:06.599546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:06.599504 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" event={"ID":"86451d4b-a607-4d8e-90ef-ab2267aa7705","Type":"ContainerStarted","Data":"4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8"} Oct 29 22:28:06.735848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:06.735813 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:28:06.736193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:06.736160 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:28:06.736269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:06.736227 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 29 22:28:07.504016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:07.503338 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 22:28:07.504280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:07.504240 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 22:28:07.736081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:07.736050 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:28:07.736457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:07.736110 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 29 22:28:08.736256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:08.736218 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:28:08.736668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:08.736277 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 29 22:28:09.522119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:09.522092 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 22:28:09.522336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:09.522157 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 22:28:09.736248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:09.736218 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:28:09.736467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:09.736442 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 29 22:28:10.537531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:10.537490 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 22:28:10.538749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:10.538730 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 22:28:10.736183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:10.736151 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:28:10.736368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:10.736207 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 29 22:28:11.735965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:11.735872 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:28:11.735965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:11.735935 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 29 22:28:12.569886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.568925 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 22:28:12.571024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.570403 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 22:28:12.572499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.571924 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 22:28:12.572826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.572808 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 22:28:12.740998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.740908 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:28:12.741950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.741922 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:28:12.743071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:12.743046 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:28:14.916115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.916078 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr"] Oct 29 22:28:14.918963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.918942 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:14.946094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.946054 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr"] Oct 29 22:28:14.984232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.983008 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"kube-root-ca.crt\"" Oct 29 22:28:14.984232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.983086 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"openshift-service-ca.crt\"" Oct 29 22:28:14.984232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.983184 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"simulator-w2-8d3632-kserve-self-signed-certs\"" Oct 29 22:28:14.985165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:14.985123 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"default-dockercfg-tlzc2\"" Oct 29 22:28:15.011180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.010878 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.011180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.010923 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.011180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.010970 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.011180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.010988 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.011180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.011024 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckpw\" (UniqueName: \"kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.011180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.011081 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.112589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.112554 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckpw\" (UniqueName: \"kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.112827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.112633 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.112827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.112668 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.112827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.112708 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.112827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.112767 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.112827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.112784 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.113266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.113237 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.113421 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.113397 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.113519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.113500 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.115200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.115169 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.116361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.116339 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.132386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.132353 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckpw\" (UniqueName: \"kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw\") pod \"simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.233949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.233849 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:15.588638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.588598 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr"] Oct 29 22:28:15.698664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.698634 5367 generic.go:356] "Generic (PLEG): container finished" podID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerID="bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648" exitCode=0 Oct 29 22:28:15.698895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.698872 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" event={"ID":"aff1f6bc-160e-4ab6-815d-c344cd149a65","Type":"ContainerDied","Data":"bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648"} Oct 29 22:28:15.699554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.699517 5367 scope.go:122] "RemoveContainer" containerID="bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648" Oct 29 22:28:15.701564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.701188 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:15.701564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.701257 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:15.701564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.701295 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:15.702182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.702134 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" event={"ID":"c34e200d-12be-48ab-917a-b2ab3a05a55a","Type":"ContainerStarted","Data":"8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869"} Oct 29 22:28:15.702182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.702164 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" event={"ID":"c34e200d-12be-48ab-917a-b2ab3a05a55a","Type":"ContainerStarted","Data":"e91883b72e55ce0b1f85b3e86a0eec91e12d1c91287ff294cb44e80f08c71d6a"} Oct 29 22:28:15.807224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:15.807177 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podStartSLOduration=1.80716206 podStartE2EDuration="1.80716206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:15.805849173 +0000 UTC m=+762.699924098" watchObservedRunningTime="2026-10-29 22:28:15.80716206 +0000 UTC m=+762.701236981" Oct 29 22:28:16.234959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:16.234929 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:16.728003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:16.727726 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" event={"ID":"aff1f6bc-160e-4ab6-815d-c344cd149a65","Type":"ContainerStarted","Data":"2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0"} Oct 29 22:28:17.161725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:17.158039 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:28:17.178722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:17.164821 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:28:17.178722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:17.164882 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 22:28:18.171230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:18.171196 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:28:18.171826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:18.171793 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 22:28:19.173728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:19.162133 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:28:19.173728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:19.162201 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 22:28:20.161727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:20.159089 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:28:20.161727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:20.159157 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 22:28:20.395722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:20.395480 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:28:21.161724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:21.158068 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:28:21.161724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:21.158135 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 22:28:21.444901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:21.444811 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:28:21.444901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:21.444873 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:22.158722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:22.158082 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:28:22.158722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:22.158150 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 22:28:22.363989 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:28:22.363949 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:28:23.163487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.163440 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:28:23.166616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.166547 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:28:23.872571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.872532 5367 generic.go:356] "Generic (PLEG): container finished" podID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerID="e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11" exitCode=0 Oct 29 22:28:23.872853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.872606 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" event={"ID":"3b7fed1d-bb2f-44e3-b1af-6043e5208157","Type":"ContainerDied","Data":"e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11"} Oct 29 22:28:23.873576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.873135 5367 scope.go:122] "RemoveContainer" containerID="e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11" Oct 29 22:28:23.874483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.874456 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:23.874553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.874520 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:23.874553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:23.874548 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:24.889079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:24.889043 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" event={"ID":"3b7fed1d-bb2f-44e3-b1af-6043e5208157","Type":"ContainerStarted","Data":"ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0"} Oct 29 22:28:25.234849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:25.234765 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:25.247943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:25.247915 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:25.887101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:25.887060 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:28:25.887393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:25.887364 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:25.887473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:25.887443 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 22:28:25.906934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:25.906898 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:28:26.837427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.837380 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj"] Oct 29 22:28:26.841450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.841419 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:26.882348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.882296 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"openshift-service-ca.crt\"" Oct 29 22:28:26.886678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.886652 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:26.886837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.886711 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 22:28:26.895961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.895927 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"simulator-w1-879596-kserve-self-signed-certs\"" Oct 29 22:28:26.895961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.895927 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"kube-root-ca.crt\"" Oct 29 22:28:26.896151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.896042 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"default-dockercfg-jks2w\"" Oct 29 22:28:26.899048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.899022 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj"] Oct 29 22:28:26.903894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.903865 5367 generic.go:356] "Generic (PLEG): container finished" podID="0fe5c579-0c5f-4404-be47-1c61db827670" containerID="4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f" exitCode=0 Oct 29 22:28:26.904033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.903939 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" event={"ID":"0fe5c579-0c5f-4404-be47-1c61db827670","Type":"ContainerDied","Data":"4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f"} Oct 29 22:28:26.904448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.904429 5367 scope.go:122] "RemoveContainer" containerID="4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f" Oct 29 22:28:26.905628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.905600 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:26.905711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.905666 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:26.905769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.905716 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:26.906380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.906357 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:26.906470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.906437 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:26.906527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.906465 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:26.906660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.906639 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:26.906717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.906682 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xf4n\" (UniqueName: \"kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:26.906759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:26.906740 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.007309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.007276 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.007749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.007324 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xf4n\" (UniqueName: \"kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.007749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.007350 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.007749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.007451 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.007749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.007519 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.007749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.007547 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.008111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.008068 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.010038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.010013 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.010780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.010758 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.011557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.011533 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.014092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.014076 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.039638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.030139 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xf4n\" (UniqueName: \"kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n\") pod \"simulator-w1-879596-kserve-f7989cf76-wtgtj\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.152765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.152662 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:27.504621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.504527 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj"] Oct 29 22:28:27.887213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.887167 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:27.887493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.887465 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 22:28:27.913343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.913304 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" event={"ID":"9cb0eb6c-2930-431f-a36d-c6f3dda9b228","Type":"ContainerStarted","Data":"8a9d94a3ec67d6063863bb8419713e2cd8cf30f9470587f8403d1b81a4d2bf63"} Oct 29 22:28:27.913547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.913363 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" event={"ID":"9cb0eb6c-2930-431f-a36d-c6f3dda9b228","Type":"ContainerStarted","Data":"406fcb040c7e7a5306e919a4dce101fd3205816bead1cdabcbb1296ffccd0d1f"} Oct 29 22:28:27.915567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.915541 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" event={"ID":"0fe5c579-0c5f-4404-be47-1c61db827670","Type":"ContainerStarted","Data":"4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841"} Oct 29 22:28:27.989341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:27.989279 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podStartSLOduration=1.989258883 podStartE2EDuration="1.989258883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:27.965557284 +0000 UTC m=+774.859632209" watchObservedRunningTime="2026-10-29 22:28:27.989258883 +0000 UTC m=+774.883333807" Oct 29 22:28:28.153208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.153127 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:28.211500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.211425 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9"] Oct 29 22:28:28.218558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.218526 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.220590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.220389 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9"] Oct 29 22:28:28.234948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.234928 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"kube-root-ca.crt\"" Oct 29 22:28:28.235129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.235117 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"openshift-service-ca.crt\"" Oct 29 22:28:28.298049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.298016 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"default-dockercfg-t54gx\"" Oct 29 22:28:28.306727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.298388 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"simulator-w3-95672c-kserve-self-signed-certs\"" Oct 29 22:28:28.323511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.323477 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.323682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.323561 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.323766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.323747 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.323864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.323848 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.324028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.323955 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvgn\" (UniqueName: \"kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.324028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.324006 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.425484 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.425557 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.425619 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.425685 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.425784 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.425878 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvgn\" (UniqueName: \"kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.426523 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.426787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.426519 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.427168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.426867 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.429032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.429005 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.429364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.429342 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.446659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.446608 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvgn\" (UniqueName: \"kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn\") pod \"simulator-w3-95672c-kserve-5957b56dbd-xpxs9\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.502730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.502087 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:28:28.502730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.502224 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:28:28.502730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.502270 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 29 22:28:28.531071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.531028 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:28.905724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.905460 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:28.905724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.905522 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 22:28:28.905724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.905640 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9"] Oct 29 22:28:28.910180 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:28:28.909445 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72a3568c_59dd_4b3c_9e3f_3e9ca926c3b8.slice/crio-48292ec89a75d44edea50bfea8fc35c49c0caebed9de9a99ef401fede4cae5db WatchSource:0}: Error finding container 48292ec89a75d44edea50bfea8fc35c49c0caebed9de9a99ef401fede4cae5db: Status 404 returned error can't find the container with id 48292ec89a75d44edea50bfea8fc35c49c0caebed9de9a99ef401fede4cae5db Oct 29 22:28:28.926749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:28.926723 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" event={"ID":"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8","Type":"ContainerStarted","Data":"48292ec89a75d44edea50bfea8fc35c49c0caebed9de9a99ef401fede4cae5db"} Oct 29 22:28:29.499145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:29.499102 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:28:29.499790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:29.499758 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 29 22:28:29.887353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:29.887308 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:29.887537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:29.887378 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 22:28:29.935068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:29.935023 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" event={"ID":"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8","Type":"ContainerStarted","Data":"27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1"} Oct 29 22:28:30.496969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.496932 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:28:30.497152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.496987 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 29 22:28:30.531901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.531856 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:30.890602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.890573 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:28:30.890975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.890959 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:28:30.891548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.891531 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 22:28:30.951428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:30.951381 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podStartSLOduration=2.951366125 podStartE2EDuration="2.951366125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:29.984943254 +0000 UTC m=+776.879018176" watchObservedRunningTime="2026-10-29 22:28:30.951366125 +0000 UTC m=+777.845441046" Oct 29 22:28:31.497842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:31.497798 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:28:31.498020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:31.497865 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 29 22:28:32.497399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:32.497353 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:28:32.497917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:32.497414 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 29 22:28:33.501292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:33.501258 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:28:33.501789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:33.501664 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:28:33.502301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:33.502281 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:28:37.153476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:37.153437 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:37.165784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:37.165758 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:38.001803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:38.001776 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:28:38.531651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:38.531612 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:38.544612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:38.544582 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:39.009630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:39.009602 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:28:40.697237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.697197 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx"] Oct 29 22:28:40.701012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.700987 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.723410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.723376 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx"] Oct 29 22:28:40.732053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.732024 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"openshift-service-ca.crt\"" Oct 29 22:28:40.732053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.732024 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"default-dockercfg-zfq69\"" Oct 29 22:28:40.732285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.732098 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"kube-root-ca.crt\"" Oct 29 22:28:40.799777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.799743 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bmc\" (UniqueName: \"kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.799944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.799785 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.799944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.799851 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.799944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.799927 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.800115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.799973 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.800159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.800116 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.818488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.818463 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"simulator-w4-4a0287-kserve-self-signed-certs\"" Oct 29 22:28:40.901383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901350 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901395 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901466 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901587 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bmc\" (UniqueName: \"kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901636 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901692 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901862 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.901988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.901972 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.902031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.902017 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.903953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.903929 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.904065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.904001 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:40.921626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:40.921603 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bmc\" (UniqueName: \"kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc\") pod \"simulator-w4-4a0287-kserve-787c996f59-7j4mx\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:41.014612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:41.014536 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:41.315312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:41.315286 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx"] Oct 29 22:28:41.316646 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:28:41.316618 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68ad178b_eb24_499b_a9a8_6f1d7e08b8b3.slice/crio-b286037618484ffd8457635a69fb6cead98d4a65fc4d85e9a3c7a061ccbc96e7 WatchSource:0}: Error finding container b286037618484ffd8457635a69fb6cead98d4a65fc4d85e9a3c7a061ccbc96e7: Status 404 returned error can't find the container with id b286037618484ffd8457635a69fb6cead98d4a65fc4d85e9a3c7a061ccbc96e7 Oct 29 22:28:42.013921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:42.013891 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" event={"ID":"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3","Type":"ContainerStarted","Data":"39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a"} Oct 29 22:28:42.014289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:42.013929 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" event={"ID":"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3","Type":"ContainerStarted","Data":"b286037618484ffd8457635a69fb6cead98d4a65fc4d85e9a3c7a061ccbc96e7"} Oct 29 22:28:42.015104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:42.015087 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:42.055432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:42.055381 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podStartSLOduration=2.055368148 podStartE2EDuration="2.055368148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:42.05386989 +0000 UTC m=+788.947944810" watchObservedRunningTime="2026-10-29 22:28:42.055368148 +0000 UTC m=+788.949443068" Oct 29 22:28:48.243323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.243290 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx"] Oct 29 22:28:48.246120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.246098 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.262031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.262009 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-879596-openshift-default-dockercfg-gkrg6\"" Oct 29 22:28:48.278679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.278654 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx"] Oct 29 22:28:48.288291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288261 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288327 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288375 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288453 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288470 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288536 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288605 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288664 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.288741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.288682 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdp2\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.389626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.389593 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.389830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.389635 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.389830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.389766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.389975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.389864 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.389975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.389893 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.389975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.389957 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390003 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390070 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390125 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390222 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390254 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdp2\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390325 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390449 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.390593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.390570 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.392222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.392205 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.392359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.392343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.409893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.409863 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.410008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.409994 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdp2\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.558603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.558581 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:48.877716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.877676 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx"] Oct 29 22:28:48.879268 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:28:48.879240 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3a0dc5_300a_47f9_925a_939895679004.slice/crio-250004dd30544ce608dbb735736ddcfecd2c6b6fa95c50256b5d15bbf6089c46 WatchSource:0}: Error finding container 250004dd30544ce608dbb735736ddcfecd2c6b6fa95c50256b5d15bbf6089c46: Status 404 returned error can't find the container with id 250004dd30544ce608dbb735736ddcfecd2c6b6fa95c50256b5d15bbf6089c46 Oct 29 22:28:48.881774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.881742 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:48.881861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.881805 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:48.881861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:48.881834 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:49.072071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:49.072034 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" event={"ID":"1f3a0dc5-300a-47f9-925a-939895679004","Type":"ContainerStarted","Data":"7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370"} Oct 29 22:28:49.072244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:49.072114 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" event={"ID":"1f3a0dc5-300a-47f9-925a-939895679004","Type":"ContainerStarted","Data":"250004dd30544ce608dbb735736ddcfecd2c6b6fa95c50256b5d15bbf6089c46"} Oct 29 22:28:49.121553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:49.121500 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podStartSLOduration=1.121486297 podStartE2EDuration="1.121486297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:49.118193609 +0000 UTC m=+796.012268533" watchObservedRunningTime="2026-10-29 22:28:49.121486297 +0000 UTC m=+796.015561218" Oct 29 22:28:49.564768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:49.560115 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:49.578911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:49.578866 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:28:49.579139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:49.578943 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 22:28:50.570722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:50.561012 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:28:50.570722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:50.561071 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 22:28:51.015123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:51.015037 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:51.027771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:51.027743 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:51.100575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:51.100544 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:28:51.562109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:51.562069 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:28:51.562418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:51.562380 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 22:28:52.332348 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:28:52.332297 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:28:52.566100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:52.565275 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:28:52.566100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:52.565331 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 22:28:53.578117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:53.578077 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:28:53.578759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:53.578686 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 22:28:54.566488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:54.566040 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:28:54.566488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:54.566113 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 22:28:55.563496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:55.563466 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:55.563926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:55.563818 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:28:55.564584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:28:55.564565 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:29:08.116470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.116436 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh"] Oct 29 22:29:08.120594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.120571 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.139818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.139791 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"default-dockercfg-b5wbl\"" Oct 29 22:29:08.152854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.152830 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"openshift-service-ca.crt\"" Oct 29 22:29:08.152854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.152861 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"kube-root-ca.crt\"" Oct 29 22:29:08.157134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.157111 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh"] Oct 29 22:29:08.231168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.231139 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.231371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.231203 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.231371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.231244 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92h9q\" (UniqueName: \"kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.231371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.231280 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.231371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.231293 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.231513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.231439 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.239829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.239812 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"premium-w0-40cf9a-kserve-self-signed-certs\"" Oct 29 22:29:08.332560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.332519 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.332819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.332786 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.332959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.332888 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.333027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.333003 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.333027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.333019 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92h9q\" (UniqueName: \"kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.333194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.333175 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.333266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.333189 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.333266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.333206 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.333857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.333837 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.336080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.336060 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.336180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.336156 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.354828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.354791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92h9q\" (UniqueName: \"kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q\") pod \"premium-w0-40cf9a-kserve-bcdb45979-5wrbh\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.459896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.459803 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:08.760522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:08.760493 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh"] Oct 29 22:29:08.761488 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:08.761463 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b7fd0ba_3560_45c9_b059_679e64e807a5.slice/crio-1ecaf3222ee16de93079e6e363a2364c4d1a85161abd34aa1d8d4ebde8258ce3 WatchSource:0}: Error finding container 1ecaf3222ee16de93079e6e363a2364c4d1a85161abd34aa1d8d4ebde8258ce3: Status 404 returned error can't find the container with id 1ecaf3222ee16de93079e6e363a2364c4d1a85161abd34aa1d8d4ebde8258ce3 Oct 29 22:29:09.208654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:09.208616 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" event={"ID":"1b7fd0ba-3560-45c9-b059-679e64e807a5","Type":"ContainerStarted","Data":"ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395"} Oct 29 22:29:09.208654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:09.208660 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" event={"ID":"1b7fd0ba-3560-45c9-b059-679e64e807a5","Type":"ContainerStarted","Data":"1ecaf3222ee16de93079e6e363a2364c4d1a85161abd34aa1d8d4ebde8258ce3"} Oct 29 22:29:09.248951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:09.248902 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podStartSLOduration=1.248888645 podStartE2EDuration="1.248888645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:09.24789778 +0000 UTC m=+816.141972706" watchObservedRunningTime="2026-10-29 22:29:09.248888645 +0000 UTC m=+816.142963565" Oct 29 22:29:09.460936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:09.460845 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:18.460345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:18.460263 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:18.472359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:18.472333 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:19.279639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:19.279605 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:29:22.330734 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:29:22.330675 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:29:33.038063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.038031 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b"] Oct 29 22:29:33.041222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.041203 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.058012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.057989 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-40cf9a-openshift-default-dockercfg-8bd6l\"" Oct 29 22:29:33.076619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.076585 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b"] Oct 29 22:29:33.107587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107552 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhx9s\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107605 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107722 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107794 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107812 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107832 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107860 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.107900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107890 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.108142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.107935 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209293 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhx9s\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209333 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209360 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209415 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209433 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209460 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209500 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209538 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209596 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.209868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.209756 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.210094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.210067 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.210335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.210306 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.210472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.210394 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.210547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.210473 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.212191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.212167 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.212309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.212166 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.231243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.231215 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.231863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.231844 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhx9s\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.370723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.370297 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-40cf9a-openshift-default-dockercfg-8bd6l\"" Oct 29 22:29:33.375673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.375587 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:33.709389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.709361 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b"] Oct 29 22:29:33.709847 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:33.709819 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1ff5d8_5462_4302_9296_fdf9a4597911.slice/crio-f3dc9c0ec940ad77786d347575e6b0008c6777294ec2c1f1a55134b96ed703d1 WatchSource:0}: Error finding container f3dc9c0ec940ad77786d347575e6b0008c6777294ec2c1f1a55134b96ed703d1: Status 404 returned error can't find the container with id f3dc9c0ec940ad77786d347575e6b0008c6777294ec2c1f1a55134b96ed703d1 Oct 29 22:29:33.712288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.712253 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:29:33.712377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.712330 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:29:33.712444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:33.712376 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:29:34.383688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:34.383655 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" event={"ID":"5f1ff5d8-5462-4302-9296-fdf9a4597911","Type":"ContainerStarted","Data":"e1e44f355bacbcffadad4311676f349667e15786a427dd8d7e965d3f298938fb"} Oct 29 22:29:34.384208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:34.383722 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" event={"ID":"5f1ff5d8-5462-4302-9296-fdf9a4597911","Type":"ContainerStarted","Data":"f3dc9c0ec940ad77786d347575e6b0008c6777294ec2c1f1a55134b96ed703d1"} Oct 29 22:29:34.461646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:34.461559 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podStartSLOduration=1.461536797 podStartE2EDuration="1.461536797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:34.443541744 +0000 UTC m=+841.337616667" watchObservedRunningTime="2026-10-29 22:29:34.461536797 +0000 UTC m=+841.355611719" Oct 29 22:29:35.378487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:35.378454 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:35.378884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:35.378860 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:29:35.379053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:35.379029 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 22:29:36.382723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:36.382132 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:29:36.382723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:36.382197 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 22:29:37.390726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:37.379225 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:29:37.390726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:37.379306 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 22:29:38.377723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.377136 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:29:38.377723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.377207 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 22:29:38.935588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.935558 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk"] Oct 29 22:29:38.938550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.938533 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:38.974400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.974368 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:29:38.974794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.974775 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:29:38.975513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.974998 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:29:38.975513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.974884 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:29:38.993407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:38.993378 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk"] Oct 29 22:29:39.051483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.051454 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.051641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.051496 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.051641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.051615 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qfb\" (UniqueName: \"kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.153245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.153213 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.153245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.153256 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.153501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.153305 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qfb\" (UniqueName: \"kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.178771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.154031 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.179985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.179268 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.179985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.179925 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qfb\" (UniqueName: \"kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb\") pod \"maas-api-revoke-sub-374802a8-5kslk\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.251172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.251088 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:39.385536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.385128 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:29:39.385536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.385188 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 22:29:39.582574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.582544 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk"] Oct 29 22:29:39.584422 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:39.584385 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b304ba5_ce09_4a09_b445_ee9407e0cf33.slice/crio-c0b99aa6414626bc6c42218a2e5cfbf12c1495f7f77d5eb711294f618daa4f2e WatchSource:0}: Error finding container c0b99aa6414626bc6c42218a2e5cfbf12c1495f7f77d5eb711294f618daa4f2e: Status 404 returned error can't find the container with id c0b99aa6414626bc6c42218a2e5cfbf12c1495f7f77d5eb711294f618daa4f2e Oct 29 22:29:39.586316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:39.586296 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:29:40.376257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:40.376209 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:29:40.376691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:40.376354 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 22:29:40.447409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:40.447368 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" event={"ID":"3b304ba5-ce09-4a09-b445-ee9407e0cf33","Type":"ContainerStarted","Data":"c0b99aa6414626bc6c42218a2e5cfbf12c1495f7f77d5eb711294f618daa4f2e"} Oct 29 22:29:41.957242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:41.957209 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:41.957887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:41.957862 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:41.958590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:41.958567 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:29:42.465766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:42.465731 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" event={"ID":"3b304ba5-ce09-4a09-b445-ee9407e0cf33","Type":"ContainerStarted","Data":"b3476cdb53e1c9af2df6a22be348944dbfe615bfd8666a57b2b6f8bb26c12fbd"} Oct 29 22:29:42.528560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:42.528439 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" podStartSLOduration=2.1657019809999998 podStartE2EDuration="4.528419623s" totalImagesPullingTime="2.362717642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:42.527623718 +0000 UTC m=+849.421698630" watchObservedRunningTime="2026-10-29 22:29:42.528419623 +0000 UTC m=+849.422494544" Oct 29 22:29:43.003210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.003173 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6"] Oct 29 22:29:43.006067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.006045 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.027215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.027184 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.027675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.027336 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.027675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.027382 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r25n\" (UniqueName: \"kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.027845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.027796 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6"] Oct 29 22:29:43.128592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.128548 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r25n\" (UniqueName: \"kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.128796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.128674 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.128897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.128877 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.129463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.129440 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.130891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.130870 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.152487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.152453 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r25n\" (UniqueName: \"kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n\") pod \"maas-api-revoke-sub-d318ddbd-b7xb6\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.347730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.345018 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:43.483745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.483405 5367 generic.go:356] "Generic (PLEG): container finished" podID="3b304ba5-ce09-4a09-b445-ee9407e0cf33" containerID="b3476cdb53e1c9af2df6a22be348944dbfe615bfd8666a57b2b6f8bb26c12fbd" exitCode=0 Oct 29 22:29:43.484258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.484201 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" event={"ID":"3b304ba5-ce09-4a09-b445-ee9407e0cf33","Type":"ContainerDied","Data":"b3476cdb53e1c9af2df6a22be348944dbfe615bfd8666a57b2b6f8bb26c12fbd"} Oct 29 22:29:43.585661 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.585632 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr"] Oct 29 22:29:43.588906 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.588888 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.611774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.610979 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr"] Oct 29 22:29:43.648497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.648469 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.648643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.648502 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4qv\" (UniqueName: \"kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.648788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.648764 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.711292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.711261 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6"] Oct 29 22:29:43.712725 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:43.712679 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaad35a0_c508_401b_b5ad_cc4374f10218.slice/crio-02232d61223c121435dd0c08932344309e240c2c0c648380b7635289f3a03c78 WatchSource:0}: Error finding container 02232d61223c121435dd0c08932344309e240c2c0c648380b7635289f3a03c78: Status 404 returned error can't find the container with id 02232d61223c121435dd0c08932344309e240c2c0c648380b7635289f3a03c78 Oct 29 22:29:43.749924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.749899 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.750010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.749997 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.750051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.750034 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4qv\" (UniqueName: \"kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.750586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.750564 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.752136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.752117 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.770763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.770738 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4qv\" (UniqueName: \"kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv\") pod \"maas-api-revoke-sub-388199dc-7fjmr\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:43.902229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:43.902204 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:44.203444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.203414 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr"] Oct 29 22:29:44.205032 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:44.205007 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838a0363_0d5a_489a_b097_10b15467be93.slice/crio-c20a507ce02467014fe828a92dec30df6f129b4c88d016f9b42a32fc70795afb WatchSource:0}: Error finding container c20a507ce02467014fe828a92dec30df6f129b4c88d016f9b42a32fc70795afb: Status 404 returned error can't find the container with id c20a507ce02467014fe828a92dec30df6f129b4c88d016f9b42a32fc70795afb Oct 29 22:29:44.490892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.490853 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" event={"ID":"caad35a0-c508-401b-b5ad-cc4374f10218","Type":"ContainerStarted","Data":"7777472665e2bb468af6492d4dd820c30e0540fc4ee68f68748b9ace73c19285"} Oct 29 22:29:44.491058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.490919 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" event={"ID":"caad35a0-c508-401b-b5ad-cc4374f10218","Type":"ContainerStarted","Data":"02232d61223c121435dd0c08932344309e240c2c0c648380b7635289f3a03c78"} Oct 29 22:29:44.492314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.492289 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" event={"ID":"838a0363-0d5a-489a-b097-10b15467be93","Type":"ContainerStarted","Data":"c20a507ce02467014fe828a92dec30df6f129b4c88d016f9b42a32fc70795afb"} Oct 29 22:29:44.539037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.538981 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" podStartSLOduration=2.538965383 podStartE2EDuration="2.538965383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:44.536720192 +0000 UTC m=+851.430795105" watchObservedRunningTime="2026-10-29 22:29:44.538965383 +0000 UTC m=+851.433040303" Oct 29 22:29:44.659544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.659510 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:44.762646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.762572 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb\" (UniqueName: \"kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb\") pod \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " Oct 29 22:29:44.762646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.762636 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca\") pod \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " Oct 29 22:29:44.762854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.762673 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca\") pod \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\" (UID: \"3b304ba5-ce09-4a09-b445-ee9407e0cf33\") " Oct 29 22:29:44.763042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.763016 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca" pod "3b304ba5-ce09-4a09-b445-ee9407e0cf33" (UID: "3b304ba5-ce09-4a09-b445-ee9407e0cf33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:44.763192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.763176 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:44.764914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.764891 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb" pod "3b304ba5-ce09-4a09-b445-ee9407e0cf33" (UID: "3b304ba5-ce09-4a09-b445-ee9407e0cf33"). InnerVolumeSpecName "kube-api-access-45qfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:44.764914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.764902 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca" pod "3b304ba5-ce09-4a09-b445-ee9407e0cf33" (UID: "3b304ba5-ce09-4a09-b445-ee9407e0cf33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:29:44.887775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.868674 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qfb\" (UniqueName: \"kubernetes.io/projected/3b304ba5-ce09-4a09-b445-ee9407e0cf33-kube-api-access-45qfb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:44.887775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:44.868724 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b304ba5-ce09-4a09-b445-ee9407e0cf33-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:45.527775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.526127 5367 generic.go:356] "Generic (PLEG): container finished" podID="caad35a0-c508-401b-b5ad-cc4374f10218" containerID="7777472665e2bb468af6492d4dd820c30e0540fc4ee68f68748b9ace73c19285" exitCode=0 Oct 29 22:29:45.527775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.526300 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" event={"ID":"caad35a0-c508-401b-b5ad-cc4374f10218","Type":"ContainerDied","Data":"7777472665e2bb468af6492d4dd820c30e0540fc4ee68f68748b9ace73c19285"} Oct 29 22:29:45.534934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.528452 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" event={"ID":"3b304ba5-ce09-4a09-b445-ee9407e0cf33","Type":"ContainerDied","Data":"c0b99aa6414626bc6c42218a2e5cfbf12c1495f7f77d5eb711294f618daa4f2e"} Oct 29 22:29:45.534934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.528499 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0b99aa6414626bc6c42218a2e5cfbf12c1495f7f77d5eb711294f618daa4f2e" Oct 29 22:29:45.534934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.528557 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk" Oct 29 22:29:45.534934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.534503 5367 generic.go:356] "Generic (PLEG): container finished" podID="838a0363-0d5a-489a-b097-10b15467be93" containerID="073e3b3a3d87a5218d3a43d60f53f74616e6314895cbda95fda6b5c23250184a" exitCode=0 Oct 29 22:29:45.534934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:45.534589 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" event={"ID":"838a0363-0d5a-489a-b097-10b15467be93","Type":"ContainerDied","Data":"073e3b3a3d87a5218d3a43d60f53f74616e6314895cbda95fda6b5c23250184a"} Oct 29 22:29:46.701093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.701068 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:46.737734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.737707 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:46.808930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.808901 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca\") pod \"caad35a0-c508-401b-b5ad-cc4374f10218\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " Oct 29 22:29:46.809107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.808953 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca\") pod \"caad35a0-c508-401b-b5ad-cc4374f10218\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " Oct 29 22:29:46.809107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.808982 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca\") pod \"838a0363-0d5a-489a-b097-10b15467be93\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " Oct 29 22:29:46.809237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809139 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv\" (UniqueName: \"kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv\") pod \"838a0363-0d5a-489a-b097-10b15467be93\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " Oct 29 22:29:46.809237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809202 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca\") pod \"838a0363-0d5a-489a-b097-10b15467be93\" (UID: \"838a0363-0d5a-489a-b097-10b15467be93\") " Oct 29 22:29:46.809331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809291 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n\" (UniqueName: \"kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n\") pod \"caad35a0-c508-401b-b5ad-cc4374f10218\" (UID: \"caad35a0-c508-401b-b5ad-cc4374f10218\") " Oct 29 22:29:46.809386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809326 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca" pod "838a0363-0d5a-489a-b097-10b15467be93" (UID: "838a0363-0d5a-489a-b097-10b15467be93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:46.809386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809323 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca" pod "caad35a0-c508-401b-b5ad-cc4374f10218" (UID: "caad35a0-c508-401b-b5ad-cc4374f10218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:46.809730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809692 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:46.809846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.809734 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a0363-0d5a-489a-b097-10b15467be93-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:46.811561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.811531 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n" pod "caad35a0-c508-401b-b5ad-cc4374f10218" (UID: "caad35a0-c508-401b-b5ad-cc4374f10218"). InnerVolumeSpecName "kube-api-access-9r25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:46.811670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.811649 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca" pod "838a0363-0d5a-489a-b097-10b15467be93" (UID: "838a0363-0d5a-489a-b097-10b15467be93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:29:46.811811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.811791 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca" pod "caad35a0-c508-401b-b5ad-cc4374f10218" (UID: "caad35a0-c508-401b-b5ad-cc4374f10218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:29:46.811879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.811827 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv" pod "838a0363-0d5a-489a-b097-10b15467be93" (UID: "838a0363-0d5a-489a-b097-10b15467be93"). InnerVolumeSpecName "kube-api-access-ts4qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:46.911200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.911172 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caad35a0-c508-401b-b5ad-cc4374f10218-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:46.911200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.911198 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4qv\" (UniqueName: \"kubernetes.io/projected/838a0363-0d5a-489a-b097-10b15467be93-kube-api-access-ts4qv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:46.911391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.911209 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a0363-0d5a-489a-b097-10b15467be93-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:46.911391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:46.911218 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r25n\" (UniqueName: \"kubernetes.io/projected/caad35a0-c508-401b-b5ad-cc4374f10218-kube-api-access-9r25n\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:47.549468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:47.549390 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" Oct 29 22:29:47.549468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:47.549430 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6" event={"ID":"caad35a0-c508-401b-b5ad-cc4374f10218","Type":"ContainerDied","Data":"02232d61223c121435dd0c08932344309e240c2c0c648380b7635289f3a03c78"} Oct 29 22:29:47.549658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:47.549501 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02232d61223c121435dd0c08932344309e240c2c0c648380b7635289f3a03c78" Oct 29 22:29:47.550841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:47.550820 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" Oct 29 22:29:47.550947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:47.550820 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr" event={"ID":"838a0363-0d5a-489a-b097-10b15467be93","Type":"ContainerDied","Data":"c20a507ce02467014fe828a92dec30df6f129b4c88d016f9b42a32fc70795afb"} Oct 29 22:29:47.550996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:47.550956 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c20a507ce02467014fe828a92dec30df6f129b4c88d016f9b42a32fc70795afb" Oct 29 22:29:52.113932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.113893 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9"] Oct 29 22:29:52.115096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115070 5367 cpu_manager.go:414] "containerMap: removing container" podUID="caad35a0-c508-401b-b5ad-cc4374f10218" containerName="revoke-keys" Oct 29 22:29:52.115096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115097 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caad35a0-c508-401b-b5ad-cc4374f10218" containerName="revoke-keys" Oct 29 22:29:52.115253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115122 5367 cpu_manager.go:414] "containerMap: removing container" podUID="838a0363-0d5a-489a-b097-10b15467be93" containerName="revoke-keys" Oct 29 22:29:52.115253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115130 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838a0363-0d5a-489a-b097-10b15467be93" containerName="revoke-keys" Oct 29 22:29:52.115253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115191 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3b304ba5-ce09-4a09-b445-ee9407e0cf33" containerName="revoke-keys" Oct 29 22:29:52.115253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115200 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b304ba5-ce09-4a09-b445-ee9407e0cf33" containerName="revoke-keys" Oct 29 22:29:52.115466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115452 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b304ba5-ce09-4a09-b445-ee9407e0cf33" containerName="revoke-keys" Oct 29 22:29:52.115512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115480 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="caad35a0-c508-401b-b5ad-cc4374f10218" containerName="revoke-keys" Oct 29 22:29:52.115512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.115501 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="838a0363-0d5a-489a-b097-10b15467be93" containerName="revoke-keys" Oct 29 22:29:52.120225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.120205 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.136856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.136817 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9"] Oct 29 22:29:52.147949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.147924 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:29:52.148104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.147963 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:29:52.148183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.147969 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:29:52.150050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.150030 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:29:52.180378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.180352 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.180475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.180387 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.180475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.180433 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gw5\" (UniqueName: \"kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.281181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.281153 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.281181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.281184 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.281397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.281238 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gw5\" (UniqueName: \"kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.281842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.281816 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.283499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.283477 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.301351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.301328 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gw5\" (UniqueName: \"kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5\") pod \"maas-api-revoke-sub-0c23eab8-ncjs9\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.330854 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:29:52.330822 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:29:52.434596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.434509 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:52.736584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:52.736555 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9"] Oct 29 22:29:52.737524 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:52.737501 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe9e46c_3a00_424a_bc25_55e500ff72b6.slice/crio-52c65bbf76f5fbd815236e8782f191ed28d0733ccf628139a1d97b2d111bbfdc WatchSource:0}: Error finding container 52c65bbf76f5fbd815236e8782f191ed28d0733ccf628139a1d97b2d111bbfdc: Status 404 returned error can't find the container with id 52c65bbf76f5fbd815236e8782f191ed28d0733ccf628139a1d97b2d111bbfdc Oct 29 22:29:53.511509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.511433 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht"] Oct 29 22:29:53.515316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.515295 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.537076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.537051 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht"] Oct 29 22:29:53.593537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.593509 5367 generic.go:356] "Generic (PLEG): container finished" podID="7fe9e46c-3a00-424a-bc25-55e500ff72b6" containerID="f0a4762dca6c2d2bdb09d17d44782b06e2dbabd030ebc27130382f6bd233b4c2" exitCode=0 Oct 29 22:29:53.593652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.593576 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" event={"ID":"7fe9e46c-3a00-424a-bc25-55e500ff72b6","Type":"ContainerDied","Data":"f0a4762dca6c2d2bdb09d17d44782b06e2dbabd030ebc27130382f6bd233b4c2"} Oct 29 22:29:53.593652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.593645 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" event={"ID":"7fe9e46c-3a00-424a-bc25-55e500ff72b6","Type":"ContainerStarted","Data":"52c65bbf76f5fbd815236e8782f191ed28d0733ccf628139a1d97b2d111bbfdc"} Oct 29 22:29:53.596117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.596093 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc792\" (UniqueName: \"kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.596229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.596141 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.596229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.596211 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.697260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.697221 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc792\" (UniqueName: \"kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.697437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.697286 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.697437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.697321 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.697993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.697971 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.699713 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.699678 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.718433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.718403 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc792\" (UniqueName: \"kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792\") pod \"maas-api-revoke-sub-99df198a-7d6ht\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:53.828897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:53.828866 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:54.132137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.132100 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht"] Oct 29 22:29:54.133401 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:54.133363 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8744b9f9_35c7_4a42_a6bd_adebcd1790c3.slice/crio-18e9a74a6c68fc120480e43504bfa866a2e3c761a855c7fb2112e087000f170e WatchSource:0}: Error finding container 18e9a74a6c68fc120480e43504bfa866a2e3c761a855c7fb2112e087000f170e: Status 404 returned error can't find the container with id 18e9a74a6c68fc120480e43504bfa866a2e3c761a855c7fb2112e087000f170e Oct 29 22:29:54.601541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.601500 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" event={"ID":"8744b9f9-35c7-4a42-a6bd-adebcd1790c3","Type":"ContainerStarted","Data":"d7ee54b96677b66e8333036656ee7caf612f0a24f974959c58fefab4d0d58366"} Oct 29 22:29:54.601974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.601569 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" event={"ID":"8744b9f9-35c7-4a42-a6bd-adebcd1790c3","Type":"ContainerStarted","Data":"18e9a74a6c68fc120480e43504bfa866a2e3c761a855c7fb2112e087000f170e"} Oct 29 22:29:54.754426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.754398 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:54.797377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.797325 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" podStartSLOduration=1.797311461 podStartE2EDuration="1.797311461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:54.663315629 +0000 UTC m=+861.557390551" watchObservedRunningTime="2026-10-29 22:29:54.797311461 +0000 UTC m=+861.691386382" Oct 29 22:29:54.811379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.811348 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca\") pod \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " Oct 29 22:29:54.811549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.811385 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5\" (UniqueName: \"kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5\") pod \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " Oct 29 22:29:54.811549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.811544 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca\") pod \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\" (UID: \"7fe9e46c-3a00-424a-bc25-55e500ff72b6\") " Oct 29 22:29:54.811810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.811785 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca" pod "7fe9e46c-3a00-424a-bc25-55e500ff72b6" (UID: "7fe9e46c-3a00-424a-bc25-55e500ff72b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:54.812089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.812067 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:54.813589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.813560 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca" pod "7fe9e46c-3a00-424a-bc25-55e500ff72b6" (UID: "7fe9e46c-3a00-424a-bc25-55e500ff72b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:29:54.813667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.813650 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5" pod "7fe9e46c-3a00-424a-bc25-55e500ff72b6" (UID: "7fe9e46c-3a00-424a-bc25-55e500ff72b6"). InnerVolumeSpecName "kube-api-access-h8gw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:54.913248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.913214 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8gw5\" (UniqueName: \"kubernetes.io/projected/7fe9e46c-3a00-424a-bc25-55e500ff72b6-kube-api-access-h8gw5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:54.913248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:54.913245 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe9e46c-3a00-424a-bc25-55e500ff72b6-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:55.620523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:55.612940 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" event={"ID":"7fe9e46c-3a00-424a-bc25-55e500ff72b6","Type":"ContainerDied","Data":"52c65bbf76f5fbd815236e8782f191ed28d0733ccf628139a1d97b2d111bbfdc"} Oct 29 22:29:55.620523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:55.613038 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c65bbf76f5fbd815236e8782f191ed28d0733ccf628139a1d97b2d111bbfdc" Oct 29 22:29:55.620523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:55.613114 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9" Oct 29 22:29:55.620523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:55.619728 5367 generic.go:356] "Generic (PLEG): container finished" podID="8744b9f9-35c7-4a42-a6bd-adebcd1790c3" containerID="d7ee54b96677b66e8333036656ee7caf612f0a24f974959c58fefab4d0d58366" exitCode=0 Oct 29 22:29:55.620523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:55.619825 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" event={"ID":"8744b9f9-35c7-4a42-a6bd-adebcd1790c3","Type":"ContainerDied","Data":"d7ee54b96677b66e8333036656ee7caf612f0a24f974959c58fefab4d0d58366"} Oct 29 22:29:56.776135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.776108 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:56.836291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.836260 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca\") pod \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " Oct 29 22:29:56.836568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.836547 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca\") pod \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " Oct 29 22:29:56.836648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.836596 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792\" (UniqueName: \"kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792\") pod \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\" (UID: \"8744b9f9-35c7-4a42-a6bd-adebcd1790c3\") " Oct 29 22:29:56.836995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.836960 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca" pod "8744b9f9-35c7-4a42-a6bd-adebcd1790c3" (UID: "8744b9f9-35c7-4a42-a6bd-adebcd1790c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:56.837129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.837082 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:56.838717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.838681 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792" pod "8744b9f9-35c7-4a42-a6bd-adebcd1790c3" (UID: "8744b9f9-35c7-4a42-a6bd-adebcd1790c3"). InnerVolumeSpecName "kube-api-access-qc792". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:56.838896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.838880 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca" pod "8744b9f9-35c7-4a42-a6bd-adebcd1790c3" (UID: "8744b9f9-35c7-4a42-a6bd-adebcd1790c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:29:56.890810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.890736 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9"] Oct 29 22:29:56.891415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.891391 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8744b9f9-35c7-4a42-a6bd-adebcd1790c3" containerName="revoke-keys" Oct 29 22:29:56.891415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.891412 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8744b9f9-35c7-4a42-a6bd-adebcd1790c3" containerName="revoke-keys" Oct 29 22:29:56.891570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.891438 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7fe9e46c-3a00-424a-bc25-55e500ff72b6" containerName="revoke-keys" Oct 29 22:29:56.891570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.891443 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe9e46c-3a00-424a-bc25-55e500ff72b6" containerName="revoke-keys" Oct 29 22:29:56.891570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.891529 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe9e46c-3a00-424a-bc25-55e500ff72b6" containerName="revoke-keys" Oct 29 22:29:56.891570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.891542 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8744b9f9-35c7-4a42-a6bd-adebcd1790c3" containerName="revoke-keys" Oct 29 22:29:56.894958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.894942 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:56.915920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.915891 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9"] Oct 29 22:29:56.938212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.938183 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc792\" (UniqueName: \"kubernetes.io/projected/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-kube-api-access-qc792\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:56.938212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:56.938211 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8744b9f9-35c7-4a42-a6bd-adebcd1790c3-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:29:57.039732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.039681 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.039940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.039767 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.039940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.039820 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99q8\" (UniqueName: \"kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.141525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.141444 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.141525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.141476 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.141525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.141501 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99q8\" (UniqueName: \"kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.142239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.142207 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.143942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.143919 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.162430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.162401 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99q8\" (UniqueName: \"kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8\") pod \"maas-api-revoke-sub-12b6331c-9wlb9\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.214276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.214255 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:29:57.517402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.517382 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9"] Oct 29 22:29:57.518357 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:29:57.518329 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296071a7_c807_410f_ad09_8a97c61ffb09.slice/crio-2a80f09283352d44f2f125304e8e2a3206c3cee0d9491c8016d104eee5c43a6f WatchSource:0}: Error finding container 2a80f09283352d44f2f125304e8e2a3206c3cee0d9491c8016d104eee5c43a6f: Status 404 returned error can't find the container with id 2a80f09283352d44f2f125304e8e2a3206c3cee0d9491c8016d104eee5c43a6f Oct 29 22:29:57.633864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.633842 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" Oct 29 22:29:57.633864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.633851 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht" event={"ID":"8744b9f9-35c7-4a42-a6bd-adebcd1790c3","Type":"ContainerDied","Data":"18e9a74a6c68fc120480e43504bfa866a2e3c761a855c7fb2112e087000f170e"} Oct 29 22:29:57.634051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.633897 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e9a74a6c68fc120480e43504bfa866a2e3c761a855c7fb2112e087000f170e" Oct 29 22:29:57.635118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:57.635096 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" event={"ID":"296071a7-c807-410f-ad09-8a97c61ffb09","Type":"ContainerStarted","Data":"2a80f09283352d44f2f125304e8e2a3206c3cee0d9491c8016d104eee5c43a6f"} Oct 29 22:29:58.686778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:58.686560 5367 generic.go:356] "Generic (PLEG): container finished" podID="296071a7-c807-410f-ad09-8a97c61ffb09" containerID="ce750159cd61be4a632f37dbdf4cd99ed0efe59ea44919d1491c04b093122ff0" exitCode=0 Oct 29 22:29:58.687265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:58.686779 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" event={"ID":"296071a7-c807-410f-ad09-8a97c61ffb09","Type":"ContainerDied","Data":"ce750159cd61be4a632f37dbdf4cd99ed0efe59ea44919d1491c04b093122ff0"} Oct 29 22:29:59.595267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.595232 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c"] Oct 29 22:29:59.599120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.599098 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.626441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.626414 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c"] Oct 29 22:29:59.694766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.694723 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nz5\" (UniqueName: \"kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.695239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.694814 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.695239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.694917 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.810383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.810283 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nz5\" (UniqueName: \"kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.810383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.810377 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.810606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.810463 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.850568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.836843 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.853628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.853500 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:29:59.929991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:29:59.929929 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nz5\" (UniqueName: \"kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5\") pod \"maas-api-revoke-sub-c41a370b-vqp6c\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:30:00.040354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.040053 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:30:00.118776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.118059 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca\") pod \"296071a7-c807-410f-ad09-8a97c61ffb09\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " Oct 29 22:30:00.118776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.118212 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca\") pod \"296071a7-c807-410f-ad09-8a97c61ffb09\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " Oct 29 22:30:00.118776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.118353 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8\" (UniqueName: \"kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8\") pod \"296071a7-c807-410f-ad09-8a97c61ffb09\" (UID: \"296071a7-c807-410f-ad09-8a97c61ffb09\") " Oct 29 22:30:00.119443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.119098 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca" pod "296071a7-c807-410f-ad09-8a97c61ffb09" (UID: "296071a7-c807-410f-ad09-8a97c61ffb09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:00.143907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.143871 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8" pod "296071a7-c807-410f-ad09-8a97c61ffb09" (UID: "296071a7-c807-410f-ad09-8a97c61ffb09"). InnerVolumeSpecName "kube-api-access-b99q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:00.144592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.144558 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca" pod "296071a7-c807-410f-ad09-8a97c61ffb09" (UID: "296071a7-c807-410f-ad09-8a97c61ffb09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:00.153398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.153376 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf"] Oct 29 22:30:00.180721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.180192 5367 cpu_manager.go:414] "containerMap: removing container" podUID="296071a7-c807-410f-ad09-8a97c61ffb09" containerName="revoke-keys" Oct 29 22:30:00.180721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.180220 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296071a7-c807-410f-ad09-8a97c61ffb09" containerName="revoke-keys" Oct 29 22:30:00.180721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.180437 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="296071a7-c807-410f-ad09-8a97c61ffb09" containerName="revoke-keys" Oct 29 22:30:00.207720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.207029 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.227721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.216873 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:30:00.244783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.237268 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99q8\" (UniqueName: \"kubernetes.io/projected/296071a7-c807-410f-ad09-8a97c61ffb09-kube-api-access-b99q8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:00.244783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.237306 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:00.244783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.237324 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296071a7-c807-410f-ad09-8a97c61ffb09-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:00.245048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.245007 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw"] Oct 29 22:30:00.347739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.347410 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vgp\" (UniqueName: \"kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.347739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.347502 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.357738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.353614 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.417795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.366752 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5"] Oct 29 22:30:00.446097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.445630 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf"] Oct 29 22:30:00.446097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.445759 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.450051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.447410 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs"] Oct 29 22:30:00.451977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.451949 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.452175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.452154 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfh6p\" (UniqueName: \"kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.452447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.452402 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.452613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.452599 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vgp\" (UniqueName: \"kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.453771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.453751 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.499760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.474882 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw"] Oct 29 22:30:00.499760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.475010 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.551316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.551286 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w"] Oct 29 22:30:00.552637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.552578 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vgp\" (UniqueName: \"kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.553517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.553437 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnxvk\" (UniqueName: \"kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.553732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.553675 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvtw\" (UniqueName: \"kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.583724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.561847 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfh6p\" (UniqueName: \"kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.583724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.561931 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.583724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.562013 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.583724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.562061 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.583724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.562865 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.707928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.655376 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfh6p\" (UniqueName: \"kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.707928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.655562 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:00.707928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.691910 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnxvk\" (UniqueName: \"kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.707928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.692011 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvtw\" (UniqueName: \"kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.710962 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.711070 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.711904 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.712893 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.716763 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5"] Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.716872 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.726667 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs"] Oct 29 22:30:00.737522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.726747 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w"] Oct 29 22:30:00.839467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.770428 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvtw\" (UniqueName: \"kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:00.844872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.844229 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnxvk\" (UniqueName: \"kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:00.870835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.854151 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:00.969830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.953815 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvgb\" (UniqueName: \"kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:00.969830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:00.954052 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:01.045742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.033932 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:01.046720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.046661 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:01.151896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.151865 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvgb\" (UniqueName: \"kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:01.152645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.152603 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:01.158717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.153830 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:01.244850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.239129 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" Oct 29 22:30:01.244850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.238985 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9" event={"ID":"296071a7-c807-410f-ad09-8a97c61ffb09","Type":"ContainerDied","Data":"2a80f09283352d44f2f125304e8e2a3206c3cee0d9491c8016d104eee5c43a6f"} Oct 29 22:30:01.244850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.239765 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a80f09283352d44f2f125304e8e2a3206c3cee0d9491c8016d104eee5c43a6f" Oct 29 22:30:01.244850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.242045 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvgb\" (UniqueName: \"kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:01.344008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.342814 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c"] Oct 29 22:30:01.484549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.484524 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:01.762183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.755783 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw"] Oct 29 22:30:01.762183 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:01.757776 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9874eb00_4e45_401c_8a09_6095aaf5db95.slice/crio-27b5af4607c62de800455335d36480f554196b051ef8a6aee54631e4c38980db WatchSource:0}: Error finding container 27b5af4607c62de800455335d36480f554196b051ef8a6aee54631e4c38980db: Status 404 returned error can't find the container with id 27b5af4607c62de800455335d36480f554196b051ef8a6aee54631e4c38980db Oct 29 22:30:01.790978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.790955 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5"] Oct 29 22:30:01.794928 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:01.794905 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a18977_e2ce_4ed5_8c9e_8d4e4aaad81a.slice/crio-3dc50960ac1d71803b65ba45e1339f7876d8f467469cddfdc9b6c345a9535667 WatchSource:0}: Error finding container 3dc50960ac1d71803b65ba45e1339f7876d8f467469cddfdc9b6c345a9535667: Status 404 returned error can't find the container with id 3dc50960ac1d71803b65ba45e1339f7876d8f467469cddfdc9b6c345a9535667 Oct 29 22:30:01.832124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.832102 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf"] Oct 29 22:30:01.836941 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:01.836911 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3a5929_e35d_4f0f_aa76_965b06b3d59e.slice/crio-643069139d8e4831d9f2b051aa090831ba1c8d89eceee1b9bb3678f5a42f2cf7 WatchSource:0}: Error finding container 643069139d8e4831d9f2b051aa090831ba1c8d89eceee1b9bb3678f5a42f2cf7: Status 404 returned error can't find the container with id 643069139d8e4831d9f2b051aa090831ba1c8d89eceee1b9bb3678f5a42f2cf7 Oct 29 22:30:01.864169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.864141 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs"] Oct 29 22:30:01.866746 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:01.866684 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6206f1a_3910_4c32_9035_e6b0b17cc908.slice/crio-8362cfb56024de0d4fee7c7fde3924daf9f67a5c30e5f8885891a6695f9861eb WatchSource:0}: Error finding container 8362cfb56024de0d4fee7c7fde3924daf9f67a5c30e5f8885891a6695f9861eb: Status 404 returned error can't find the container with id 8362cfb56024de0d4fee7c7fde3924daf9f67a5c30e5f8885891a6695f9861eb Oct 29 22:30:01.897891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:01.897796 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w"] Oct 29 22:30:02.248312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.248276 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" event={"ID":"0e979399-1cdc-404f-93c6-edfb5958da08","Type":"ContainerStarted","Data":"dc1e27dddb2202083eeb02698296d925c862fc7b263bd9e9a80194d9552da4ce"} Oct 29 22:30:02.248480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.248335 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" event={"ID":"0e979399-1cdc-404f-93c6-edfb5958da08","Type":"ContainerStarted","Data":"5ff1d7454551a6b89997cf9d2d01afd278d4a482fb68585e1789ac0676ff65e5"} Oct 29 22:30:02.249638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.249610 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" event={"ID":"da68764e-1255-4def-8318-a90f4b8b9fa8","Type":"ContainerStarted","Data":"1c88e875add3323715b1529a78e1dd9fe1106332ae1a0f2ea347ac6fcd263c9d"} Oct 29 22:30:02.250818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.250790 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" event={"ID":"a6206f1a-3910-4c32-9035-e6b0b17cc908","Type":"ContainerStarted","Data":"8362cfb56024de0d4fee7c7fde3924daf9f67a5c30e5f8885891a6695f9861eb"} Oct 29 22:30:02.252294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.252274 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerStarted","Data":"6ba3c0f87342b5b0aa531203dd6b28d461948b65df68888a49b19b545c6e10dc"} Oct 29 22:30:02.252384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.252310 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerStarted","Data":"643069139d8e4831d9f2b051aa090831ba1c8d89eceee1b9bb3678f5a42f2cf7"} Oct 29 22:30:02.253484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.253462 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" event={"ID":"9874eb00-4e45-401c-8a09-6095aaf5db95","Type":"ContainerStarted","Data":"27b5af4607c62de800455335d36480f554196b051ef8a6aee54631e4c38980db"} Oct 29 22:30:02.254927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.254902 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" event={"ID":"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a","Type":"ContainerStarted","Data":"3dc50960ac1d71803b65ba45e1339f7876d8f467469cddfdc9b6c345a9535667"} Oct 29 22:30:02.300599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.300545 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" podStartSLOduration=3.300520825 podStartE2EDuration="3.300520825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:02.297210489 +0000 UTC m=+869.191285407" watchObservedRunningTime="2026-10-29 22:30:02.300520825 +0000 UTC m=+869.194595744" Oct 29 22:30:02.349834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:02.349742 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" podStartSLOduration=2.349729806 podStartE2EDuration="2.349729806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:02.348288756 +0000 UTC m=+869.242363681" watchObservedRunningTime="2026-10-29 22:30:02.349729806 +0000 UTC m=+869.243804725" Oct 29 22:30:02.903111 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:02.903076 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3a5929_e35d_4f0f_aa76_965b06b3d59e.slice/crio-6ba3c0f87342b5b0aa531203dd6b28d461948b65df68888a49b19b545c6e10dc.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:30:03.267565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.267526 5367 generic.go:356] "Generic (PLEG): container finished" podID="0e979399-1cdc-404f-93c6-edfb5958da08" containerID="dc1e27dddb2202083eeb02698296d925c862fc7b263bd9e9a80194d9552da4ce" exitCode=0 Oct 29 22:30:03.267833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.267620 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" event={"ID":"0e979399-1cdc-404f-93c6-edfb5958da08","Type":"ContainerDied","Data":"dc1e27dddb2202083eeb02698296d925c862fc7b263bd9e9a80194d9552da4ce"} Oct 29 22:30:03.269468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.269440 5367 generic.go:356] "Generic (PLEG): container finished" podID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerID="220d40ddff68344277131b1814cd7f2d7d13750344ca7b893b1b81e64b1d29ba" exitCode=60 Oct 29 22:30:03.269599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.269530 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" event={"ID":"da68764e-1255-4def-8318-a90f4b8b9fa8","Type":"ContainerDied","Data":"220d40ddff68344277131b1814cd7f2d7d13750344ca7b893b1b81e64b1d29ba"} Oct 29 22:30:03.269851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.269828 5367 scope.go:122] "RemoveContainer" containerID="220d40ddff68344277131b1814cd7f2d7d13750344ca7b893b1b81e64b1d29ba" Oct 29 22:30:03.271405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.271384 5367 generic.go:356] "Generic (PLEG): container finished" podID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerID="7821f95e296505eb4ede8450d94b042b1d41da21a315c70e2692a8e8e5f340b2" exitCode=60 Oct 29 22:30:03.271511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.271489 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" event={"ID":"a6206f1a-3910-4c32-9035-e6b0b17cc908","Type":"ContainerDied","Data":"7821f95e296505eb4ede8450d94b042b1d41da21a315c70e2692a8e8e5f340b2"} Oct 29 22:30:03.271689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.271672 5367 scope.go:122] "RemoveContainer" containerID="7821f95e296505eb4ede8450d94b042b1d41da21a315c70e2692a8e8e5f340b2" Oct 29 22:30:03.273626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.273599 5367 generic.go:356] "Generic (PLEG): container finished" podID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerID="6ba3c0f87342b5b0aa531203dd6b28d461948b65df68888a49b19b545c6e10dc" exitCode=60 Oct 29 22:30:03.273779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.273749 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerDied","Data":"6ba3c0f87342b5b0aa531203dd6b28d461948b65df68888a49b19b545c6e10dc"} Oct 29 22:30:03.273995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.273981 5367 scope.go:122] "RemoveContainer" containerID="6ba3c0f87342b5b0aa531203dd6b28d461948b65df68888a49b19b545c6e10dc" Oct 29 22:30:03.280253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.280231 5367 generic.go:356] "Generic (PLEG): container finished" podID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerID="5ff985e9fc2af38be604389ee5f5d0bc660cd61b68944243fa48036e0b05bc73" exitCode=60 Oct 29 22:30:03.280433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.280416 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" event={"ID":"9874eb00-4e45-401c-8a09-6095aaf5db95","Type":"ContainerDied","Data":"5ff985e9fc2af38be604389ee5f5d0bc660cd61b68944243fa48036e0b05bc73"} Oct 29 22:30:03.280939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.280922 5367 scope.go:122] "RemoveContainer" containerID="5ff985e9fc2af38be604389ee5f5d0bc660cd61b68944243fa48036e0b05bc73" Oct 29 22:30:03.286777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.286755 5367 generic.go:356] "Generic (PLEG): container finished" podID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerID="5eff637c1db4bcbd7c5b18ae048be3fc9801dc501ea3a627f096631c6bf8af97" exitCode=60 Oct 29 22:30:03.286888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.286813 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" event={"ID":"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a","Type":"ContainerDied","Data":"5eff637c1db4bcbd7c5b18ae048be3fc9801dc501ea3a627f096631c6bf8af97"} Oct 29 22:30:03.287026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.287012 5367 scope.go:122] "RemoveContainer" containerID="5eff637c1db4bcbd7c5b18ae048be3fc9801dc501ea3a627f096631c6bf8af97" Oct 29 22:30:03.638016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.637987 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x"] Oct 29 22:30:03.642991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.642968 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.680865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.680841 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x"] Oct 29 22:30:03.722962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.722919 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.722978 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723076 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723127 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w56\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723228 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723292 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723340 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723410 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.723509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.723490 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824309 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824351 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824376 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824431 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824509 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824541 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824562 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824601 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w56\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824631 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824680 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824906 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.824970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.824933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.825092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.825071 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.825289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.825268 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.826440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.826420 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.826834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.826815 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.859437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.859408 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.860880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.860855 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w56\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:03.956863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:03.956042 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:04.296009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.295982 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" event={"ID":"da68764e-1255-4def-8318-a90f4b8b9fa8","Type":"ContainerStarted","Data":"8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37"} Oct 29 22:30:04.298195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.298173 5367 generic.go:356] "Generic (PLEG): container finished" podID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerID="d0d638692738e4a57e94d84eaf04eae549982d51cd5a0dcf05fc7c703127d0fc" exitCode=60 Oct 29 22:30:04.298316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.298259 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" event={"ID":"a6206f1a-3910-4c32-9035-e6b0b17cc908","Type":"ContainerDied","Data":"d0d638692738e4a57e94d84eaf04eae549982d51cd5a0dcf05fc7c703127d0fc"} Oct 29 22:30:04.298384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.298315 5367 scope.go:122] "RemoveContainer" containerID="7821f95e296505eb4ede8450d94b042b1d41da21a315c70e2692a8e8e5f340b2" Oct 29 22:30:04.298603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.298585 5367 scope.go:122] "RemoveContainer" containerID="d0d638692738e4a57e94d84eaf04eae549982d51cd5a0dcf05fc7c703127d0fc" Oct 29 22:30:04.298926 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:04.298902 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs_odh-ai-gateway-infra(a6206f1a-3910-4c32-9035-e6b0b17cc908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" Oct 29 22:30:04.300502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.300477 5367 generic.go:356] "Generic (PLEG): container finished" podID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerID="03b5a98c7bc6bd34256d75e43a65d2462052ab30784ea125acf5177477e929be" exitCode=60 Oct 29 22:30:04.300601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.300556 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerDied","Data":"03b5a98c7bc6bd34256d75e43a65d2462052ab30784ea125acf5177477e929be"} Oct 29 22:30:04.300876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.300856 5367 scope.go:122] "RemoveContainer" containerID="03b5a98c7bc6bd34256d75e43a65d2462052ab30784ea125acf5177477e929be" Oct 29 22:30:04.301156 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:04.301135 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf_odh-ai-gateway-infra(0f3a5929-e35d-4f0f-aa76-965b06b3d59e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" Oct 29 22:30:04.301780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.301759 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x"] Oct 29 22:30:04.302441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.302423 5367 generic.go:356] "Generic (PLEG): container finished" podID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerID="a7dcf0932a026b7ee8a1a5c22dad6924d5355262566097a777448e714c39ed8b" exitCode=60 Oct 29 22:30:04.302544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.302506 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" event={"ID":"9874eb00-4e45-401c-8a09-6095aaf5db95","Type":"ContainerDied","Data":"a7dcf0932a026b7ee8a1a5c22dad6924d5355262566097a777448e714c39ed8b"} Oct 29 22:30:04.302771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.302750 5367 scope.go:122] "RemoveContainer" containerID="a7dcf0932a026b7ee8a1a5c22dad6924d5355262566097a777448e714c39ed8b" Oct 29 22:30:04.302972 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:04.302948 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw_odh-ai-gateway-infra(9874eb00-4e45-401c-8a09-6095aaf5db95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" Oct 29 22:30:04.304485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.304455 5367 generic.go:356] "Generic (PLEG): container finished" podID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerID="387c219f55ba62e30443fb803b51758ff603b263a5a87772b0ffe20a0b4c161d" exitCode=60 Oct 29 22:30:04.304610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.304545 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" event={"ID":"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a","Type":"ContainerDied","Data":"387c219f55ba62e30443fb803b51758ff603b263a5a87772b0ffe20a0b4c161d"} Oct 29 22:30:04.304761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.304744 5367 scope.go:122] "RemoveContainer" containerID="387c219f55ba62e30443fb803b51758ff603b263a5a87772b0ffe20a0b4c161d" Oct 29 22:30:04.304992 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:04.304952 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5_odh-ai-gateway-infra(87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" Oct 29 22:30:04.320602 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:04.320579 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d26976f_bf8d_40aa_8d2b_3fe83a092ed0.slice/crio-0d1994888b502c69ad1c6cd6da924cb1308794800a7f113e0563e92875cb4b85 WatchSource:0}: Error finding container 0d1994888b502c69ad1c6cd6da924cb1308794800a7f113e0563e92875cb4b85: Status 404 returned error can't find the container with id 0d1994888b502c69ad1c6cd6da924cb1308794800a7f113e0563e92875cb4b85 Oct 29 22:30:04.322837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.322809 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:04.322923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.322878 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:04.322923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.322909 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:04.346406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.346359 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" podStartSLOduration=4.346345209 podStartE2EDuration="4.346345209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:04.342913082 +0000 UTC m=+871.236988004" watchObservedRunningTime="2026-10-29 22:30:04.346345209 +0000 UTC m=+871.240420128" Oct 29 22:30:04.376942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.376915 5367 scope.go:122] "RemoveContainer" containerID="6ba3c0f87342b5b0aa531203dd6b28d461948b65df68888a49b19b545c6e10dc" Oct 29 22:30:04.398040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.398016 5367 scope.go:122] "RemoveContainer" containerID="5ff985e9fc2af38be604389ee5f5d0bc660cd61b68944243fa48036e0b05bc73" Oct 29 22:30:04.416344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.416213 5367 scope.go:122] "RemoveContainer" containerID="5eff637c1db4bcbd7c5b18ae048be3fc9801dc501ea3a627f096631c6bf8af97" Oct 29 22:30:04.452086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.452055 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:30:04.532804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.532772 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5\" (UniqueName: \"kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5\") pod \"0e979399-1cdc-404f-93c6-edfb5958da08\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " Oct 29 22:30:04.532976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.532827 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca\") pod \"0e979399-1cdc-404f-93c6-edfb5958da08\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " Oct 29 22:30:04.532976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.532884 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca\") pod \"0e979399-1cdc-404f-93c6-edfb5958da08\" (UID: \"0e979399-1cdc-404f-93c6-edfb5958da08\") " Oct 29 22:30:04.533190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.533163 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca" pod "0e979399-1cdc-404f-93c6-edfb5958da08" (UID: "0e979399-1cdc-404f-93c6-edfb5958da08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:04.533396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.533380 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:04.534904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.534881 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5" pod "0e979399-1cdc-404f-93c6-edfb5958da08" (UID: "0e979399-1cdc-404f-93c6-edfb5958da08"). InnerVolumeSpecName "kube-api-access-j8nz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:04.535024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.534977 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca" pod "0e979399-1cdc-404f-93c6-edfb5958da08" (UID: "0e979399-1cdc-404f-93c6-edfb5958da08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:04.634833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.634797 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e979399-1cdc-404f-93c6-edfb5958da08-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:04.634833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:04.634833 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8nz5\" (UniqueName: \"kubernetes.io/projected/0e979399-1cdc-404f-93c6-edfb5958da08-kube-api-access-j8nz5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:05.312481 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.312452 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" Oct 29 22:30:05.312978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.312451 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c" event={"ID":"0e979399-1cdc-404f-93c6-edfb5958da08","Type":"ContainerDied","Data":"5ff1d7454551a6b89997cf9d2d01afd278d4a482fb68585e1789ac0676ff65e5"} Oct 29 22:30:05.312978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.312635 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff1d7454551a6b89997cf9d2d01afd278d4a482fb68585e1789ac0676ff65e5" Oct 29 22:30:05.314694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.314665 5367 generic.go:356] "Generic (PLEG): container finished" podID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerID="8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37" exitCode=60 Oct 29 22:30:05.314831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.314802 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" event={"ID":"da68764e-1255-4def-8318-a90f4b8b9fa8","Type":"ContainerDied","Data":"8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37"} Oct 29 22:30:05.314899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.314849 5367 scope.go:122] "RemoveContainer" containerID="220d40ddff68344277131b1814cd7f2d7d13750344ca7b893b1b81e64b1d29ba" Oct 29 22:30:05.315248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.315232 5367 scope.go:122] "RemoveContainer" containerID="8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37" Oct 29 22:30:05.315493 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:05.315474 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w_odh-ai-gateway-infra(da68764e-1255-4def-8318-a90f4b8b9fa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" Oct 29 22:30:05.318116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.318095 5367 scope.go:122] "RemoveContainer" containerID="d0d638692738e4a57e94d84eaf04eae549982d51cd5a0dcf05fc7c703127d0fc" Oct 29 22:30:05.318358 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:05.318336 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs_odh-ai-gateway-infra(a6206f1a-3910-4c32-9035-e6b0b17cc908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" Oct 29 22:30:05.320225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.320207 5367 scope.go:122] "RemoveContainer" containerID="03b5a98c7bc6bd34256d75e43a65d2462052ab30784ea125acf5177477e929be" Oct 29 22:30:05.320446 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:05.320423 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf_odh-ai-gateway-infra(0f3a5929-e35d-4f0f-aa76-965b06b3d59e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" Oct 29 22:30:05.321960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.321944 5367 scope.go:122] "RemoveContainer" containerID="a7dcf0932a026b7ee8a1a5c22dad6924d5355262566097a777448e714c39ed8b" Oct 29 22:30:05.322127 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:05.322110 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw_odh-ai-gateway-infra(9874eb00-4e45-401c-8a09-6095aaf5db95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" Oct 29 22:30:05.323476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.323462 5367 scope.go:122] "RemoveContainer" containerID="387c219f55ba62e30443fb803b51758ff603b263a5a87772b0ffe20a0b4c161d" Oct 29 22:30:05.323657 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:05.323636 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5_odh-ai-gateway-infra(87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" Oct 29 22:30:05.324747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.324691 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" event={"ID":"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0","Type":"ContainerStarted","Data":"89896b35ebc64bbe7f78e715ce6665e848e6e03631b82cbb749e1811ad4eb8e4"} Oct 29 22:30:05.324839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.324760 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" event={"ID":"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0","Type":"ContainerStarted","Data":"0d1994888b502c69ad1c6cd6da924cb1308794800a7f113e0563e92875cb4b85"} Oct 29 22:30:05.643258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.643205 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podStartSLOduration=2.643190722 podStartE2EDuration="2.643190722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:05.639493349 +0000 UTC m=+872.533568271" watchObservedRunningTime="2026-10-29 22:30:05.643190722 +0000 UTC m=+872.537265643" Oct 29 22:30:05.959786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.957987 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:05.963724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.963028 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:30:05.963724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:05.963082 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:30:06.292586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.289962 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg"] Oct 29 22:30:06.292586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.290809 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0e979399-1cdc-404f-93c6-edfb5958da08" containerName="revoke-keys" Oct 29 22:30:06.292586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.290830 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e979399-1cdc-404f-93c6-edfb5958da08" containerName="revoke-keys" Oct 29 22:30:06.292586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.291029 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e979399-1cdc-404f-93c6-edfb5958da08" containerName="revoke-keys" Oct 29 22:30:06.297734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.296586 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.318906 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.318881 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:06.321626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.321603 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg"] Oct 29 22:30:06.337194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.337169 5367 scope.go:122] "RemoveContainer" containerID="8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37" Oct 29 22:30:06.337524 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:06.337497 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w_odh-ai-gateway-infra(da68764e-1255-4def-8318-a90f4b8b9fa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" Oct 29 22:30:06.382253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.382217 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.382446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.382294 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf6w\" (UniqueName: \"kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.382446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.382329 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.483580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.483524 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.483813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.483639 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf6w\" (UniqueName: \"kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.483813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.483687 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.484680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.484653 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.486208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.486186 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.508617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.508590 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf6w\" (UniqueName: \"kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w\") pod \"maas-api-revoke-sub-fca3013a-w48fg\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.622494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.618995 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:06.992775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.985875 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:30:06.992939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.992813 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:30:06.995923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:06.995852 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg"] Oct 29 22:30:07.304293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.304247 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc"] Oct 29 22:30:07.309799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.309773 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.330371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.330348 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc"] Oct 29 22:30:07.347233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.347193 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" event={"ID":"abc3f248-554f-4b40-ac9f-9f36c2c049e9","Type":"ContainerStarted","Data":"17275efb93781bfd5c612e04c18f1b1619e68cc51f18b5af6cdf4fa572598f7e"} Oct 29 22:30:07.404867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.404833 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5rn\" (UniqueName: \"kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.404979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.404955 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.405127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.405110 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.506288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.506252 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.506456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.506351 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5rn\" (UniqueName: \"kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.506456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.506411 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.507105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.507084 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.508572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.508553 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.535365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.535338 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5rn\" (UniqueName: \"kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn\") pod \"maas-api-revoke-sub-8e2e67c2-c4pgc\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.628161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.622873 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:07.960544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.960058 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:30:07.960544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.960123 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:30:07.993692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:07.993666 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc"] Oct 29 22:30:07.994942 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:07.994913 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e166181_5ba6_442f_aa45_8a4b98c13f0d.slice/crio-1b96df271b2e6c8c3d489508f40154c83f6d1436d4dad7c767117e07aca94736 WatchSource:0}: Error finding container 1b96df271b2e6c8c3d489508f40154c83f6d1436d4dad7c767117e07aca94736: Status 404 returned error can't find the container with id 1b96df271b2e6c8c3d489508f40154c83f6d1436d4dad7c767117e07aca94736 Oct 29 22:30:08.379985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:08.379950 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" event={"ID":"0e166181-5ba6-442f-aa45-8a4b98c13f0d","Type":"ContainerStarted","Data":"1b96df271b2e6c8c3d489508f40154c83f6d1436d4dad7c767117e07aca94736"} Oct 29 22:30:08.381667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:08.381642 5367 generic.go:356] "Generic (PLEG): container finished" podID="abc3f248-554f-4b40-ac9f-9f36c2c049e9" containerID="fc0cd451ebd471630f00e5a6fee92c4fcdb233bd0a78c63d734959e50f8f4f1e" exitCode=0 Oct 29 22:30:08.381749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:08.381673 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" event={"ID":"abc3f248-554f-4b40-ac9f-9f36c2c049e9","Type":"ContainerDied","Data":"fc0cd451ebd471630f00e5a6fee92c4fcdb233bd0a78c63d734959e50f8f4f1e"} Oct 29 22:30:08.979111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:08.979072 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:30:08.979283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:08.979179 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:30:09.390642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.390596 5367 generic.go:356] "Generic (PLEG): container finished" podID="0e166181-5ba6-442f-aa45-8a4b98c13f0d" containerID="63f10639558dc00d25920faa7e68b780a4a918071bf3face9a94a307d6375f61" exitCode=0 Oct 29 22:30:09.391149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.390667 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" event={"ID":"0e166181-5ba6-442f-aa45-8a4b98c13f0d","Type":"ContainerDied","Data":"63f10639558dc00d25920faa7e68b780a4a918071bf3face9a94a307d6375f61"} Oct 29 22:30:09.581707 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.581672 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:09.631577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.631546 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca\") pod \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " Oct 29 22:30:09.631760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.631610 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w\" (UniqueName: \"kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w\") pod \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " Oct 29 22:30:09.631760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.631683 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca\") pod \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\" (UID: \"abc3f248-554f-4b40-ac9f-9f36c2c049e9\") " Oct 29 22:30:09.632370 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.632338 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca" pod "abc3f248-554f-4b40-ac9f-9f36c2c049e9" (UID: "abc3f248-554f-4b40-ac9f-9f36c2c049e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:09.634096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.634071 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w" pod "abc3f248-554f-4b40-ac9f-9f36c2c049e9" (UID: "abc3f248-554f-4b40-ac9f-9f36c2c049e9"). InnerVolumeSpecName "kube-api-access-dkf6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:09.634096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.634087 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca" pod "abc3f248-554f-4b40-ac9f-9f36c2c049e9" (UID: "abc3f248-554f-4b40-ac9f-9f36c2c049e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:09.733049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.733018 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:09.733049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.733045 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkf6w\" (UniqueName: \"kubernetes.io/projected/abc3f248-554f-4b40-ac9f-9f36c2c049e9-kube-api-access-dkf6w\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:09.733049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.733056 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc3f248-554f-4b40-ac9f-9f36c2c049e9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:09.964776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.963716 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:30:09.964776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:09.963796 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:30:10.400426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.400392 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" Oct 29 22:30:10.400863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.400421 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg" event={"ID":"abc3f248-554f-4b40-ac9f-9f36c2c049e9","Type":"ContainerDied","Data":"17275efb93781bfd5c612e04c18f1b1619e68cc51f18b5af6cdf4fa572598f7e"} Oct 29 22:30:10.400863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.400494 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17275efb93781bfd5c612e04c18f1b1619e68cc51f18b5af6cdf4fa572598f7e" Oct 29 22:30:10.550582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.550555 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:10.642001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.641965 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca\") pod \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " Oct 29 22:30:10.642172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.642154 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca\") pod \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " Oct 29 22:30:10.642334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.642316 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn\" (UniqueName: \"kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn\") pod \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\" (UID: \"0e166181-5ba6-442f-aa45-8a4b98c13f0d\") " Oct 29 22:30:10.642447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.642415 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca" pod "0e166181-5ba6-442f-aa45-8a4b98c13f0d" (UID: "0e166181-5ba6-442f-aa45-8a4b98c13f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:10.642878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.642852 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:10.644341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.644316 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca" pod "0e166181-5ba6-442f-aa45-8a4b98c13f0d" (UID: "0e166181-5ba6-442f-aa45-8a4b98c13f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:10.644942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.644918 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn" pod "0e166181-5ba6-442f-aa45-8a4b98c13f0d" (UID: "0e166181-5ba6-442f-aa45-8a4b98c13f0d"). InnerVolumeSpecName "kube-api-access-mb5rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:10.744430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.744400 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e166181-5ba6-442f-aa45-8a4b98c13f0d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:10.744430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.744429 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb5rn\" (UniqueName: \"kubernetes.io/projected/0e166181-5ba6-442f-aa45-8a4b98c13f0d-kube-api-access-mb5rn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:10.963881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.963804 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:10.964289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.964270 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:10.965031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:10.965009 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:30:11.408017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:11.407992 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" Oct 29 22:30:11.408411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:11.408030 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc" event={"ID":"0e166181-5ba6-442f-aa45-8a4b98c13f0d","Type":"ContainerDied","Data":"1b96df271b2e6c8c3d489508f40154c83f6d1436d4dad7c767117e07aca94736"} Oct 29 22:30:11.408411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:11.408076 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b96df271b2e6c8c3d489508f40154c83f6d1436d4dad7c767117e07aca94736" Oct 29 22:30:15.232722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:15.229279 5367 scope.go:122] "RemoveContainer" containerID="03b5a98c7bc6bd34256d75e43a65d2462052ab30784ea125acf5177477e929be" Oct 29 22:30:16.219723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:16.219308 5367 scope.go:122] "RemoveContainer" containerID="387c219f55ba62e30443fb803b51758ff603b263a5a87772b0ffe20a0b4c161d" Oct 29 22:30:16.222258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:16.220173 5367 scope.go:122] "RemoveContainer" containerID="8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37" Oct 29 22:30:16.222258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:16.220832 5367 scope.go:122] "RemoveContainer" containerID="a7dcf0932a026b7ee8a1a5c22dad6924d5355262566097a777448e714c39ed8b" Oct 29 22:30:16.492781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:16.492232 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerStarted","Data":"a1825340854b17b89ea9dec06ce45365edcb61088d9836312e3a4e8a168f7096"} Oct 29 22:30:17.500434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.500405 5367 generic.go:356] "Generic (PLEG): container finished" podID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerID="3f75b5ee6c27597de168292ecbcb271ae55413d61217f6021c818ff6f5caba39" exitCode=60 Oct 29 22:30:17.500770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.500476 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" event={"ID":"9874eb00-4e45-401c-8a09-6095aaf5db95","Type":"ContainerDied","Data":"3f75b5ee6c27597de168292ecbcb271ae55413d61217f6021c818ff6f5caba39"} Oct 29 22:30:17.500770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.500542 5367 scope.go:122] "RemoveContainer" containerID="a7dcf0932a026b7ee8a1a5c22dad6924d5355262566097a777448e714c39ed8b" Oct 29 22:30:17.500938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.500922 5367 scope.go:122] "RemoveContainer" containerID="3f75b5ee6c27597de168292ecbcb271ae55413d61217f6021c818ff6f5caba39" Oct 29 22:30:17.501227 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:17.501203 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw_odh-ai-gateway-infra(9874eb00-4e45-401c-8a09-6095aaf5db95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" Oct 29 22:30:17.502412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.502394 5367 generic.go:356] "Generic (PLEG): container finished" podID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerID="559f0c6f84c95a3d56a442c32a5dbcb1f28a1182da039f1f29d1bcec6b3e8ab6" exitCode=60 Oct 29 22:30:17.502485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.502463 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" event={"ID":"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a","Type":"ContainerDied","Data":"559f0c6f84c95a3d56a442c32a5dbcb1f28a1182da039f1f29d1bcec6b3e8ab6"} Oct 29 22:30:17.502893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.502879 5367 scope.go:122] "RemoveContainer" containerID="559f0c6f84c95a3d56a442c32a5dbcb1f28a1182da039f1f29d1bcec6b3e8ab6" Oct 29 22:30:17.503154 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:17.503136 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5_odh-ai-gateway-infra(87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" Oct 29 22:30:17.504299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.504281 5367 generic.go:356] "Generic (PLEG): container finished" podID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerID="9ced8c032d087e1b10bbb46378a2fc7add1c4785c3a42b5435fc545b775a493b" exitCode=60 Oct 29 22:30:17.504380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.504357 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" event={"ID":"da68764e-1255-4def-8318-a90f4b8b9fa8","Type":"ContainerDied","Data":"9ced8c032d087e1b10bbb46378a2fc7add1c4785c3a42b5435fc545b775a493b"} Oct 29 22:30:17.504679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.504664 5367 scope.go:122] "RemoveContainer" containerID="9ced8c032d087e1b10bbb46378a2fc7add1c4785c3a42b5435fc545b775a493b" Oct 29 22:30:17.504971 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:17.504952 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w_odh-ai-gateway-infra(da68764e-1255-4def-8318-a90f4b8b9fa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" Oct 29 22:30:17.506174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.506155 5367 generic.go:356] "Generic (PLEG): container finished" podID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerID="a1825340854b17b89ea9dec06ce45365edcb61088d9836312e3a4e8a168f7096" exitCode=60 Oct 29 22:30:17.506248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.506204 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerDied","Data":"a1825340854b17b89ea9dec06ce45365edcb61088d9836312e3a4e8a168f7096"} Oct 29 22:30:17.506472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.506460 5367 scope.go:122] "RemoveContainer" containerID="a1825340854b17b89ea9dec06ce45365edcb61088d9836312e3a4e8a168f7096" Oct 29 22:30:17.506736 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:17.506713 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf_odh-ai-gateway-infra(0f3a5929-e35d-4f0f-aa76-965b06b3d59e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" Oct 29 22:30:17.557711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.557672 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf"] Oct 29 22:30:17.581530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.581502 5367 scope.go:122] "RemoveContainer" containerID="387c219f55ba62e30443fb803b51758ff603b263a5a87772b0ffe20a0b4c161d" Oct 29 22:30:17.599848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.599818 5367 scope.go:122] "RemoveContainer" containerID="8d6d5601f1f915bb5846297b059301549555c29388bde4b7a4ea1ceaf8e71a37" Oct 29 22:30:17.615899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.615878 5367 scope.go:122] "RemoveContainer" containerID="03b5a98c7bc6bd34256d75e43a65d2462052ab30784ea125acf5177477e929be" Oct 29 22:30:17.734341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.734297 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh"] Oct 29 22:30:17.734935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.734917 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0e166181-5ba6-442f-aa45-8a4b98c13f0d" containerName="revoke-keys" Oct 29 22:30:17.734935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.734934 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e166181-5ba6-442f-aa45-8a4b98c13f0d" containerName="revoke-keys" Oct 29 22:30:17.735057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.734949 5367 cpu_manager.go:414] "containerMap: removing container" podUID="abc3f248-554f-4b40-ac9f-9f36c2c049e9" containerName="revoke-keys" Oct 29 22:30:17.735057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.734955 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc3f248-554f-4b40-ac9f-9f36c2c049e9" containerName="revoke-keys" Oct 29 22:30:17.735118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.735065 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e166181-5ba6-442f-aa45-8a4b98c13f0d" containerName="revoke-keys" Oct 29 22:30:17.735118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.735074 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc3f248-554f-4b40-ac9f-9f36c2c049e9" containerName="revoke-keys" Oct 29 22:30:17.740202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.740182 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.758891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.758852 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:17.762773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.762750 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh"] Oct 29 22:30:17.764990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.764965 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.765119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.765002 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.765119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.765028 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xsb\" (UniqueName: \"kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.866049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.865952 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.866049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.866000 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.866049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.866040 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xsb\" (UniqueName: \"kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.866685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.866665 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.868368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.868345 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:17.888504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:17.888480 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xsb\" (UniqueName: \"kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb\") pod \"maas-api-revoke-sub-2d263071-5jbgh\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:18.053862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.053831 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:18.215152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.215077 5367 scope.go:122] "RemoveContainer" containerID="d0d638692738e4a57e94d84eaf04eae549982d51cd5a0dcf05fc7c703127d0fc" Oct 29 22:30:18.358568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.358511 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh"] Oct 29 22:30:18.358918 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:18.358893 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c16e75e_807e_41b8_9911_5e007362a202.slice/crio-82c3a8b7f5c6808e88f387c713d6fbcb4725e4c18aac3c7c6f41d97e2822e747 WatchSource:0}: Error finding container 82c3a8b7f5c6808e88f387c713d6fbcb4725e4c18aac3c7c6f41d97e2822e747: Status 404 returned error can't find the container with id 82c3a8b7f5c6808e88f387c713d6fbcb4725e4c18aac3c7c6f41d97e2822e747 Oct 29 22:30:18.516730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.516673 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" event={"ID":"0c16e75e-807e-41b8-9911-5e007362a202","Type":"ContainerStarted","Data":"82c3a8b7f5c6808e88f387c713d6fbcb4725e4c18aac3c7c6f41d97e2822e747"} Oct 29 22:30:18.596519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.595308 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw"] Oct 29 22:30:18.652220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.650115 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w"] Oct 29 22:30:18.674767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.673065 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:18.698726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.697817 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5"] Oct 29 22:30:18.759319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.759287 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:18.778021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.777216 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca\") pod \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " Oct 29 22:30:18.778021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.777282 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp\" (UniqueName: \"kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp\") pod \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\" (UID: \"0f3a5929-e35d-4f0f-aa76-965b06b3d59e\") " Oct 29 22:30:18.778021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.777949 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca" pod "0f3a5929-e35d-4f0f-aa76-965b06b3d59e" (UID: "0f3a5929-e35d-4f0f-aa76-965b06b3d59e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:18.780486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.780462 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp" pod "0f3a5929-e35d-4f0f-aa76-965b06b3d59e" (UID: "0f3a5929-e35d-4f0f-aa76-965b06b3d59e"). InnerVolumeSpecName "kube-api-access-j2vgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:18.817301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.817273 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:18.878996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.878972 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p\" (UniqueName: \"kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p\") pod \"9874eb00-4e45-401c-8a09-6095aaf5db95\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " Oct 29 22:30:18.879137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.879069 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb\" (UniqueName: \"kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb\") pod \"da68764e-1255-4def-8318-a90f4b8b9fa8\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " Oct 29 22:30:18.879578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.879550 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca\") pod \"da68764e-1255-4def-8318-a90f4b8b9fa8\" (UID: \"da68764e-1255-4def-8318-a90f4b8b9fa8\") " Oct 29 22:30:18.879690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.879673 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca\") pod \"9874eb00-4e45-401c-8a09-6095aaf5db95\" (UID: \"9874eb00-4e45-401c-8a09-6095aaf5db95\") " Oct 29 22:30:18.882787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.880769 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca" pod "da68764e-1255-4def-8318-a90f4b8b9fa8" (UID: "da68764e-1255-4def-8318-a90f4b8b9fa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:18.882787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.882144 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68764e-1255-4def-8318-a90f4b8b9fa8-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.882787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.882168 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.882787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.882182 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vgp\" (UniqueName: \"kubernetes.io/projected/0f3a5929-e35d-4f0f-aa76-965b06b3d59e-kube-api-access-j2vgp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.883069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.882965 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca" pod "9874eb00-4e45-401c-8a09-6095aaf5db95" (UID: "9874eb00-4e45-401c-8a09-6095aaf5db95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:18.883069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.883007 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:18.891449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.891416 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p" pod "9874eb00-4e45-401c-8a09-6095aaf5db95" (UID: "9874eb00-4e45-401c-8a09-6095aaf5db95"). InnerVolumeSpecName "kube-api-access-rfh6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:18.892557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.892526 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb" pod "da68764e-1255-4def-8318-a90f4b8b9fa8" (UID: "da68764e-1255-4def-8318-a90f4b8b9fa8"). InnerVolumeSpecName "kube-api-access-bxvgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:18.982625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.982548 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca\") pod \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " Oct 29 22:30:18.982625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.982606 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw\" (UniqueName: \"kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw\") pod \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\" (UID: \"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a\") " Oct 29 22:30:18.982914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.982893 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca" pod "87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" (UID: "87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:18.983173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.983150 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9874eb00-4e45-401c-8a09-6095aaf5db95-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.983173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.983170 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfh6p\" (UniqueName: \"kubernetes.io/projected/9874eb00-4e45-401c-8a09-6095aaf5db95-kube-api-access-rfh6p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.983340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.983185 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvgb\" (UniqueName: \"kubernetes.io/projected/da68764e-1255-4def-8318-a90f4b8b9fa8-kube-api-access-bxvgb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.983340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.983197 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:18.984510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:18.984491 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw" pod "87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" (UID: "87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a"). InnerVolumeSpecName "kube-api-access-rsvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:19.084411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.084380 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvtw\" (UniqueName: \"kubernetes.io/projected/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a-kube-api-access-rsvtw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:19.456491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.456461 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9"] Oct 29 22:30:19.456997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.456982 5367 cpu_manager.go:414] "containerMap: removing container" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.456997 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457011 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457017 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457027 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457032 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457045 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457050 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457057 5367 cpu_manager.go:414] "containerMap: removing container" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457061 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457069 5367 cpu_manager.go:414] "containerMap: removing container" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457073 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457079 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457083 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457092 5367 cpu_manager.go:414] "containerMap: removing container" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457096 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457117 5367 cpu_manager.go:414] "containerMap: removing container" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457121 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457205 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457214 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457221 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457230 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457237 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457244 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457249 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457254 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457260 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457266 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:19.457329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.457272 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" containerName="cleanup" Oct 29 22:30:19.460598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.460580 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.491153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.491068 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9"] Oct 29 22:30:19.531107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.531077 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" Oct 29 22:30:19.531676 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.531079 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w" event={"ID":"da68764e-1255-4def-8318-a90f4b8b9fa8","Type":"ContainerDied","Data":"1c88e875add3323715b1529a78e1dd9fe1106332ae1a0f2ea347ac6fcd263c9d"} Oct 29 22:30:19.531676 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.531213 5367 scope.go:122] "RemoveContainer" containerID="9ced8c032d087e1b10bbb46378a2fc7add1c4785c3a42b5435fc545b775a493b" Oct 29 22:30:19.534200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.533944 5367 generic.go:356] "Generic (PLEG): container finished" podID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerID="64e6226daec8788ca392de6131c31019f05ba56870e11dbdd7a4b7866d25b2ae" exitCode=60 Oct 29 22:30:19.534200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.534042 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" event={"ID":"a6206f1a-3910-4c32-9035-e6b0b17cc908","Type":"ContainerDied","Data":"64e6226daec8788ca392de6131c31019f05ba56870e11dbdd7a4b7866d25b2ae"} Oct 29 22:30:19.534374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.534358 5367 scope.go:122] "RemoveContainer" containerID="64e6226daec8788ca392de6131c31019f05ba56870e11dbdd7a4b7866d25b2ae" Oct 29 22:30:19.534635 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:19.534617 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs_odh-ai-gateway-infra(a6206f1a-3910-4c32-9035-e6b0b17cc908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" Oct 29 22:30:19.536495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.536475 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" event={"ID":"0f3a5929-e35d-4f0f-aa76-965b06b3d59e","Type":"ContainerDied","Data":"643069139d8e4831d9f2b051aa090831ba1c8d89eceee1b9bb3678f5a42f2cf7"} Oct 29 22:30:19.536596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.536482 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf" Oct 29 22:30:19.538172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.538156 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" Oct 29 22:30:19.538256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.538168 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw" event={"ID":"9874eb00-4e45-401c-8a09-6095aaf5db95","Type":"ContainerDied","Data":"27b5af4607c62de800455335d36480f554196b051ef8a6aee54631e4c38980db"} Oct 29 22:30:19.540056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.540038 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" Oct 29 22:30:19.540139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.540059 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5" event={"ID":"87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a","Type":"ContainerDied","Data":"3dc50960ac1d71803b65ba45e1339f7876d8f467469cddfdc9b6c345a9535667"} Oct 29 22:30:19.541845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.541821 5367 generic.go:356] "Generic (PLEG): container finished" podID="0c16e75e-807e-41b8-9911-5e007362a202" containerID="81e01cc495b5296a8ebf10ca5ca6dc5d5c13d16dbead5309afd9f9f58e2dde9d" exitCode=0 Oct 29 22:30:19.541938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.541874 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" event={"ID":"0c16e75e-807e-41b8-9911-5e007362a202","Type":"ContainerDied","Data":"81e01cc495b5296a8ebf10ca5ca6dc5d5c13d16dbead5309afd9f9f58e2dde9d"} Oct 29 22:30:19.543807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.543790 5367 scope.go:122] "RemoveContainer" containerID="d0d638692738e4a57e94d84eaf04eae549982d51cd5a0dcf05fc7c703127d0fc" Oct 29 22:30:19.555999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.555980 5367 scope.go:122] "RemoveContainer" containerID="a1825340854b17b89ea9dec06ce45365edcb61088d9836312e3a4e8a168f7096" Oct 29 22:30:19.564302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.564285 5367 scope.go:122] "RemoveContainer" containerID="3f75b5ee6c27597de168292ecbcb271ae55413d61217f6021c818ff6f5caba39" Oct 29 22:30:19.572520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.572492 5367 scope.go:122] "RemoveContainer" containerID="559f0c6f84c95a3d56a442c32a5dbcb1f28a1182da039f1f29d1bcec6b3e8ab6" Oct 29 22:30:19.592878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.592855 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.593004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.592887 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.593130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.593109 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhn4c\" (UniqueName: \"kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.635689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.635664 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf"] Oct 29 22:30:19.640849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.640827 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888550-9wcgf"] Oct 29 22:30:19.680728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.680693 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw"] Oct 29 22:30:19.684588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.684568 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888550-qnfxw"] Oct 29 22:30:19.694455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.694429 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhn4c\" (UniqueName: \"kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.694570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.694483 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.694570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.694534 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.695214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.695192 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.696723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.696690 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.719529 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.719510 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhn4c\" (UniqueName: \"kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c\") pod \"maas-api-revoke-sub-20e3f44c-d5xp9\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:19.724763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.724740 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w"] Oct 29 22:30:19.729672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.729652 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888550-vw99w"] Oct 29 22:30:19.762321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.762272 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5"] Oct 29 22:30:19.768255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.768236 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888550-9srl5"] Oct 29 22:30:19.796347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:19.796331 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:20.102368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.102340 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9"] Oct 29 22:30:20.103412 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:20.103386 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c0557_8c59_4020_881c_51feda119490.slice/crio-2592f2dd6643de095c2ac43b2f3491a4e06719d4e619e9079ebad5765d165fff WatchSource:0}: Error finding container 2592f2dd6643de095c2ac43b2f3491a4e06719d4e619e9079ebad5765d165fff: Status 404 returned error can't find the container with id 2592f2dd6643de095c2ac43b2f3491a4e06719d4e619e9079ebad5765d165fff Oct 29 22:30:20.554760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.554729 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" event={"ID":"d48c0557-8c59-4020-881c-51feda119490","Type":"ContainerStarted","Data":"edfbe109ec0a3785e081f6bbc272b7859c268a1de4227826cd252b61a06631ea"} Oct 29 22:30:20.555221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.554779 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" event={"ID":"d48c0557-8c59-4020-881c-51feda119490","Type":"ContainerStarted","Data":"2592f2dd6643de095c2ac43b2f3491a4e06719d4e619e9079ebad5765d165fff"} Oct 29 22:30:20.599343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.599290 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" podStartSLOduration=1.599275797 podStartE2EDuration="1.599275797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:20.598790318 +0000 UTC m=+887.492865239" watchObservedRunningTime="2026-10-29 22:30:20.599275797 +0000 UTC m=+887.493350718" Oct 29 22:30:20.617495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.617436 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs"] Oct 29 22:30:20.711285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.711253 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:20.778348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.778315 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:20.808187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.808165 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca\") pod \"0c16e75e-807e-41b8-9911-5e007362a202\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " Oct 29 22:30:20.808318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.808283 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb\" (UniqueName: \"kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb\") pod \"0c16e75e-807e-41b8-9911-5e007362a202\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " Oct 29 22:30:20.808362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.808316 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca\") pod \"0c16e75e-807e-41b8-9911-5e007362a202\" (UID: \"0c16e75e-807e-41b8-9911-5e007362a202\") " Oct 29 22:30:20.808783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.808748 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca" pod "0c16e75e-807e-41b8-9911-5e007362a202" (UID: "0c16e75e-807e-41b8-9911-5e007362a202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:20.810059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.810038 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca" pod "0c16e75e-807e-41b8-9911-5e007362a202" (UID: "0c16e75e-807e-41b8-9911-5e007362a202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:20.810251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.810230 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb" pod "0c16e75e-807e-41b8-9911-5e007362a202" (UID: "0c16e75e-807e-41b8-9911-5e007362a202"). InnerVolumeSpecName "kube-api-access-d6xsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:20.909411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909386 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca\") pod \"a6206f1a-3910-4c32-9035-e6b0b17cc908\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " Oct 29 22:30:20.909537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909490 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk\" (UniqueName: \"kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk\") pod \"a6206f1a-3910-4c32-9035-e6b0b17cc908\" (UID: \"a6206f1a-3910-4c32-9035-e6b0b17cc908\") " Oct 29 22:30:20.909680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909658 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca" pod "a6206f1a-3910-4c32-9035-e6b0b17cc908" (UID: "a6206f1a-3910-4c32-9035-e6b0b17cc908"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:20.909852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909838 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:20.909889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909857 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xsb\" (UniqueName: \"kubernetes.io/projected/0c16e75e-807e-41b8-9911-5e007362a202-kube-api-access-d6xsb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:20.909889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909871 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16e75e-807e-41b8-9911-5e007362a202-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:20.909889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.909880 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6206f1a-3910-4c32-9035-e6b0b17cc908-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:20.911433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:20.911408 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk" pod "a6206f1a-3910-4c32-9035-e6b0b17cc908" (UID: "a6206f1a-3910-4c32-9035-e6b0b17cc908"). InnerVolumeSpecName "kube-api-access-rnxvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:21.011555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.011524 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnxvk\" (UniqueName: \"kubernetes.io/projected/a6206f1a-3910-4c32-9035-e6b0b17cc908-kube-api-access-rnxvk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.221775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.221688 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" path="/var/lib/kubelet/pods/0f3a5929-e35d-4f0f-aa76-965b06b3d59e/volumes" Oct 29 22:30:21.222162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.222148 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a" path="/var/lib/kubelet/pods/87a18977-e2ce-4ed5-8c9e-8d4e4aaad81a/volumes" Oct 29 22:30:21.222566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.222552 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" path="/var/lib/kubelet/pods/9874eb00-4e45-401c-8a09-6095aaf5db95/volumes" Oct 29 22:30:21.223266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.223248 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" path="/var/lib/kubelet/pods/da68764e-1255-4def-8318-a90f4b8b9fa8/volumes" Oct 29 22:30:21.562506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.562458 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" Oct 29 22:30:21.562506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.562472 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs" event={"ID":"a6206f1a-3910-4c32-9035-e6b0b17cc908","Type":"ContainerDied","Data":"8362cfb56024de0d4fee7c7fde3924daf9f67a5c30e5f8885891a6695f9861eb"} Oct 29 22:30:21.563072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.562563 5367 scope.go:122] "RemoveContainer" containerID="64e6226daec8788ca392de6131c31019f05ba56870e11dbdd7a4b7866d25b2ae" Oct 29 22:30:21.564355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.564325 5367 generic.go:356] "Generic (PLEG): container finished" podID="d48c0557-8c59-4020-881c-51feda119490" containerID="edfbe109ec0a3785e081f6bbc272b7859c268a1de4227826cd252b61a06631ea" exitCode=0 Oct 29 22:30:21.564464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.564436 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" event={"ID":"d48c0557-8c59-4020-881c-51feda119490","Type":"ContainerDied","Data":"edfbe109ec0a3785e081f6bbc272b7859c268a1de4227826cd252b61a06631ea"} Oct 29 22:30:21.565947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.565928 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" event={"ID":"0c16e75e-807e-41b8-9911-5e007362a202","Type":"ContainerDied","Data":"82c3a8b7f5c6808e88f387c713d6fbcb4725e4c18aac3c7c6f41d97e2822e747"} Oct 29 22:30:21.566070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.565947 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh" Oct 29 22:30:21.566070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.565962 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c3a8b7f5c6808e88f387c713d6fbcb4725e4c18aac3c7c6f41d97e2822e747" Oct 29 22:30:21.675477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.675445 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs"] Oct 29 22:30:21.678668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:21.678643 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888550-rbbxs"] Oct 29 22:30:22.112176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112146 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp"] Oct 29 22:30:22.112690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112676 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:22.112748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112693 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3a5929-e35d-4f0f-aa76-965b06b3d59e" containerName="cleanup" Oct 29 22:30:22.112748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112726 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.112748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112732 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.112748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112738 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.112748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112743 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112757 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112761 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112775 5367 cpu_manager.go:414] "containerMap: removing container" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112780 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da68764e-1255-4def-8318-a90f4b8b9fa8" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112788 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0c16e75e-807e-41b8-9911-5e007362a202" containerName="revoke-keys" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112792 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c16e75e-807e-41b8-9911-5e007362a202" containerName="revoke-keys" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112803 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112807 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:22.112889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112886 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c16e75e-807e-41b8-9911-5e007362a202" containerName="revoke-keys" Oct 29 22:30:22.113132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112895 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9874eb00-4e45-401c-8a09-6095aaf5db95" containerName="cleanup" Oct 29 22:30:22.113132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112905 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.113132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.112909 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:22.118049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.118023 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.139580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.139555 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp"] Oct 29 22:30:22.148150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.148128 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"distinct-w1-879596-kserve-self-signed-certs\"" Oct 29 22:30:22.226807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.226777 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vm6\" (UniqueName: \"kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.226935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.226914 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.226980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.226958 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.227015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.226998 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.227120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.227101 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.227166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.227143 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328109 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328197 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328246 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328295 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vm6\" (UniqueName: \"kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328417 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328467 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.328944 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.329183 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.332176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.329405 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.344180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.339138 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.344832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.344763 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.358751 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:22.356507 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:30:22.358751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.358213 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vm6\" (UniqueName: \"kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6\") pod \"distinct-w1-879596-kserve-76cf6899bd-jzfjp\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.431901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.431832 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:22.730195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.730173 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:22.733324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.733301 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp"] Oct 29 22:30:22.733660 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:22.733636 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod378e0f62_b817_47b8_ade5_40585173dc5d.slice/crio-40b62567f81bdde0571421317085e5ee19cf0fa0d6e6794bdbab452249331051 WatchSource:0}: Error finding container 40b62567f81bdde0571421317085e5ee19cf0fa0d6e6794bdbab452249331051: Status 404 returned error can't find the container with id 40b62567f81bdde0571421317085e5ee19cf0fa0d6e6794bdbab452249331051 Oct 29 22:30:22.834576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.834546 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca\") pod \"d48c0557-8c59-4020-881c-51feda119490\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " Oct 29 22:30:22.834728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.834684 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c\" (UniqueName: \"kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c\") pod \"d48c0557-8c59-4020-881c-51feda119490\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " Oct 29 22:30:22.834778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.834748 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca\") pod \"d48c0557-8c59-4020-881c-51feda119490\" (UID: \"d48c0557-8c59-4020-881c-51feda119490\") " Oct 29 22:30:22.835057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.835031 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca" pod "d48c0557-8c59-4020-881c-51feda119490" (UID: "d48c0557-8c59-4020-881c-51feda119490"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:22.835141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.835127 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c0557-8c59-4020-881c-51feda119490-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.836760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.836739 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca" pod "d48c0557-8c59-4020-881c-51feda119490" (UID: "d48c0557-8c59-4020-881c-51feda119490"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:22.836840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.836778 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c" pod "d48c0557-8c59-4020-881c-51feda119490" (UID: "d48c0557-8c59-4020-881c-51feda119490"). InnerVolumeSpecName "kube-api-access-mhn4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:22.936176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.936137 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhn4c\" (UniqueName: \"kubernetes.io/projected/d48c0557-8c59-4020-881c-51feda119490-kube-api-access-mhn4c\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.936176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:22.936167 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c0557-8c59-4020-881c-51feda119490-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:23.223106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.223073 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" path="/var/lib/kubelet/pods/a6206f1a-3910-4c32-9035-e6b0b17cc908/volumes" Oct 29 22:30:23.580122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.580099 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" Oct 29 22:30:23.580297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.580140 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9" event={"ID":"d48c0557-8c59-4020-881c-51feda119490","Type":"ContainerDied","Data":"2592f2dd6643de095c2ac43b2f3491a4e06719d4e619e9079ebad5765d165fff"} Oct 29 22:30:23.580297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.580182 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2592f2dd6643de095c2ac43b2f3491a4e06719d4e619e9079ebad5765d165fff" Oct 29 22:30:23.581848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.581819 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" event={"ID":"378e0f62-b817-47b8-ade5-40585173dc5d","Type":"ContainerStarted","Data":"a6ee479461cc4acc37ab912c0142b8a90bb01ff3580d0c3d2bb453c14643b413"} Oct 29 22:30:23.581986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.581854 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" event={"ID":"378e0f62-b817-47b8-ade5-40585173dc5d","Type":"ContainerStarted","Data":"40b62567f81bdde0571421317085e5ee19cf0fa0d6e6794bdbab452249331051"} Oct 29 22:30:23.630189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:23.630139 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podStartSLOduration=1.630127815 podStartE2EDuration="1.630127815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:23.62662494 +0000 UTC m=+890.520699873" watchObservedRunningTime="2026-10-29 22:30:23.630127815 +0000 UTC m=+890.524202734" Oct 29 22:30:24.432800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:24.432769 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:28.031953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.031921 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n"] Oct 29 22:30:28.032462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.032445 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d48c0557-8c59-4020-881c-51feda119490" containerName="revoke-keys" Oct 29 22:30:28.032516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.032464 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48c0557-8c59-4020-881c-51feda119490" containerName="revoke-keys" Oct 29 22:30:28.032564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.032548 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6206f1a-3910-4c32-9035-e6b0b17cc908" containerName="cleanup" Oct 29 22:30:28.032564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.032557 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48c0557-8c59-4020-881c-51feda119490" containerName="revoke-keys" Oct 29 22:30:28.044313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.044285 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.058456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.058425 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n"] Oct 29 22:30:28.066634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.066608 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:28.066783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.066615 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:30:28.073225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.073204 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:30:28.073225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.073209 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:30:28.116640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.116605 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.116788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.116648 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.116846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.116797 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b7p2\" (UniqueName: \"kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.217448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.217413 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b7p2\" (UniqueName: \"kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.217604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.217513 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.217604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.217539 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.218224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.218202 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.219880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.219854 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.237449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.237428 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b7p2\" (UniqueName: \"kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2\") pod \"maas-api-revoke-sub-7e23d743-6kn4n\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.357643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.357608 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:28.660167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.660138 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n"] Oct 29 22:30:28.660406 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:28.660377 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ce4bae_2082_46cb_a5ee_b1350c010d17.slice/crio-f23e2f34b5bf81c59195b88f43120b2162ab798e04770a03faa72275561b661c WatchSource:0}: Error finding container f23e2f34b5bf81c59195b88f43120b2162ab798e04770a03faa72275561b661c: Status 404 returned error can't find the container with id f23e2f34b5bf81c59195b88f43120b2162ab798e04770a03faa72275561b661c Oct 29 22:30:28.673277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:28.673249 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" event={"ID":"d4ce4bae-2082-46cb-a5ee-b1350c010d17","Type":"ContainerStarted","Data":"f23e2f34b5bf81c59195b88f43120b2162ab798e04770a03faa72275561b661c"} Oct 29 22:30:29.681479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:29.681444 5367 generic.go:356] "Generic (PLEG): container finished" podID="d4ce4bae-2082-46cb-a5ee-b1350c010d17" containerID="82d4420d6a0ce3de80acf858afe741fccb1b1cda92bf2125efb9639e403d66c2" exitCode=0 Oct 29 22:30:29.681898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:29.681527 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" event={"ID":"d4ce4bae-2082-46cb-a5ee-b1350c010d17","Type":"ContainerDied","Data":"82d4420d6a0ce3de80acf858afe741fccb1b1cda92bf2125efb9639e403d66c2"} Oct 29 22:30:30.826915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.826892 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:30.947999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.947920 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca\") pod \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " Oct 29 22:30:30.948148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.948119 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2\" (UniqueName: \"kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2\") pod \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " Oct 29 22:30:30.948201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.948163 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca\") pod \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\" (UID: \"d4ce4bae-2082-46cb-a5ee-b1350c010d17\") " Oct 29 22:30:30.948546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.948515 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca" pod "d4ce4bae-2082-46cb-a5ee-b1350c010d17" (UID: "d4ce4bae-2082-46cb-a5ee-b1350c010d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:30.949993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.949970 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca" pod "d4ce4bae-2082-46cb-a5ee-b1350c010d17" (UID: "d4ce4bae-2082-46cb-a5ee-b1350c010d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:30.950153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:30.950134 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2" pod "d4ce4bae-2082-46cb-a5ee-b1350c010d17" (UID: "d4ce4bae-2082-46cb-a5ee-b1350c010d17"). InnerVolumeSpecName "kube-api-access-5b7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:31.049513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:31.049481 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b7p2\" (UniqueName: \"kubernetes.io/projected/d4ce4bae-2082-46cb-a5ee-b1350c010d17-kube-api-access-5b7p2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:31.049513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:31.049511 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:31.049674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:31.049521 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce4bae-2082-46cb-a5ee-b1350c010d17-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:31.695853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:31.695819 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" Oct 29 22:30:31.695853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:31.695855 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n" event={"ID":"d4ce4bae-2082-46cb-a5ee-b1350c010d17","Type":"ContainerDied","Data":"f23e2f34b5bf81c59195b88f43120b2162ab798e04770a03faa72275561b661c"} Oct 29 22:30:31.695853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:31.695900 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f23e2f34b5bf81c59195b88f43120b2162ab798e04770a03faa72275561b661c" Oct 29 22:30:32.432648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:32.432611 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:32.444816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:32.444791 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:32.713262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:32.713186 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:30:33.829958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.829929 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc"] Oct 29 22:30:33.830547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.830529 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d4ce4bae-2082-46cb-a5ee-b1350c010d17" containerName="revoke-keys" Oct 29 22:30:33.830547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.830547 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ce4bae-2082-46cb-a5ee-b1350c010d17" containerName="revoke-keys" Oct 29 22:30:33.830669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.830660 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ce4bae-2082-46cb-a5ee-b1350c010d17" containerName="revoke-keys" Oct 29 22:30:33.835670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.835654 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.843357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.843291 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc"] Oct 29 22:30:33.846989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.846974 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.853929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.853907 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-95672c-openshift-default-dockercfg-hp48t\"" Oct 29 22:30:33.858239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.858216 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc"] Oct 29 22:30:33.872032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.872012 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc"] Oct 29 22:30:33.886039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886018 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886057 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886175 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886213 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886246 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886279 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886346 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886381 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886481 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzcm5\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886515 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886545 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886620 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886665 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886787 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886818 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.886871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886858 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886879 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdlq\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.886980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.886922 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.987751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987715 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcm5\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.987751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987749 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.987982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987768 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.987982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987791 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.987982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987813 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.987982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987865 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.987982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987893 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.987982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987955 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.987988 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdlq\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988013 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988044 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988089 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988131 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988155 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988194 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988244 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988251 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988314 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988361 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988396 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988408 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988514 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.988666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988541 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.989039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988684 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.989039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988816 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.989039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.988968 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.989199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.989170 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.989482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.989458 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.990629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.990544 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:33.990760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.990738 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.991019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.991000 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:33.991079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:33.991055 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:34.009305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.009279 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdlq\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:34.009409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.009331 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:34.009409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.009346 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcm5\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:34.009485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.009462 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:34.149377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.149289 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:34.170844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.170818 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:34.464001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.463973 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc"] Oct 29 22:30:34.464490 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:34.464467 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819c85fe_d774_4609_8084_199ee2718352.slice/crio-24fa8933b3433cdf74f49b9ea99675e22d2dd5b9ac03151972b647e617dbcc3a WatchSource:0}: Error finding container 24fa8933b3433cdf74f49b9ea99675e22d2dd5b9ac03151972b647e617dbcc3a: Status 404 returned error can't find the container with id 24fa8933b3433cdf74f49b9ea99675e22d2dd5b9ac03151972b647e617dbcc3a Oct 29 22:30:34.466954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.466921 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:34.467032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.466981 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:34.467032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.467013 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:34.500116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.500092 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc"] Oct 29 22:30:34.500232 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:34.500207 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee2d684_e4f5_42cb_91fb_0ec0c893c219.slice/crio-fda1a2716f3aa3b3efcf4657ac7ac91ac142a7845d64c92892cfa2f435bc94f2 WatchSource:0}: Error finding container fda1a2716f3aa3b3efcf4657ac7ac91ac142a7845d64c92892cfa2f435bc94f2: Status 404 returned error can't find the container with id fda1a2716f3aa3b3efcf4657ac7ac91ac142a7845d64c92892cfa2f435bc94f2 Oct 29 22:30:34.502378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.502352 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:34.502457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.502410 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:34.502457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.502441 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:34.728812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.728727 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" event={"ID":"5ee2d684-e4f5-42cb-91fb-0ec0c893c219","Type":"ContainerStarted","Data":"6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993"} Oct 29 22:30:34.728812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.728789 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" event={"ID":"5ee2d684-e4f5-42cb-91fb-0ec0c893c219","Type":"ContainerStarted","Data":"fda1a2716f3aa3b3efcf4657ac7ac91ac142a7845d64c92892cfa2f435bc94f2"} Oct 29 22:30:34.731577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.731544 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" event={"ID":"819c85fe-d774-4609-8084-199ee2718352","Type":"ContainerStarted","Data":"9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072"} Oct 29 22:30:34.731739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.731609 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" event={"ID":"819c85fe-d774-4609-8084-199ee2718352","Type":"ContainerStarted","Data":"24fa8933b3433cdf74f49b9ea99675e22d2dd5b9ac03151972b647e617dbcc3a"} Oct 29 22:30:34.776477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.775799 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podStartSLOduration=1.775781772 podStartE2EDuration="1.775781772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:34.77526185 +0000 UTC m=+901.669336770" watchObservedRunningTime="2026-10-29 22:30:34.775781772 +0000 UTC m=+901.669856693" Oct 29 22:30:34.826717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:34.826253 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podStartSLOduration=1.8262384950000001 podStartE2EDuration="1.826238495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:34.825733434 +0000 UTC m=+901.719808354" watchObservedRunningTime="2026-10-29 22:30:34.826238495 +0000 UTC m=+901.720313416" Oct 29 22:30:35.150455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:35.150418 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:35.152198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:35.152177 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:35.152297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:35.152228 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:35.171184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:35.171155 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:35.172314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:35.172295 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:35.172376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:35.172337 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:36.149761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:36.149724 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:36.150045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:36.149794 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:36.179723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:36.178109 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:36.179723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:36.178188 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:37.150346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:37.150310 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:37.150528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:37.150375 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:37.183724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:37.171518 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:37.183724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:37.171569 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:38.150572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:38.150095 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:38.150572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:38.150156 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:38.184757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:38.176671 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:38.184757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:38.176752 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:39.152741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:39.150021 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:39.152741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:39.150092 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:39.173723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:39.172022 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:39.173723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:39.172076 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:40.150575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:40.150544 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:40.150999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:40.150598 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:40.174090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:40.174054 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:40.174864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:40.174836 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:41.150051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:41.150020 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:30:41.150231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:41.150076 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 29 22:30:41.171343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:41.171309 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:30:41.171668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:41.171374 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:30:42.153614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:42.153582 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:42.153945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:42.153924 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:42.154685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:42.154664 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 22:30:42.175002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:42.174982 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:42.175312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:42.175225 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:42.175986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:42.175970 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:30:48.432775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.431811 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j"] Oct 29 22:30:48.449599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.449445 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j"] Oct 29 22:30:48.449599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.449604 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.545740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.545686 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.545919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.545830 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.545919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.545860 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.546021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.545932 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.546021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.545991 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.546151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.546125 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxbf\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.546285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.546167 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.546285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.546266 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.546399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.546312 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647483 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647561 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647599 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647642 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647752 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647838 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxbf\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.647877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647871 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647915 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647932 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.647954 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.648190 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.648227 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.648314 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.648716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.648682 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.649973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.649949 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.650179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.650163 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.667309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.667287 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxbf\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.667467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.667447 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.704443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.704390 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9"] Oct 29 22:30:48.708925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.708908 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.725383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.725361 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:48.725498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.725486 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:30:48.725616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.725600 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:30:48.725788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.725773 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:30:48.728096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.728071 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9"] Oct 29 22:30:48.748871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.748845 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjnx\" (UniqueName: \"kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.749021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.748906 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.749021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.749010 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.769908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.769881 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:48.850304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.849843 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjnx\" (UniqueName: \"kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.850304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.849907 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.850304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.849949 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.851007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.850981 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.852119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.852101 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:48.871289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:48.871263 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjnx\" (UniqueName: \"kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx\") pod \"maas-api-revoke-sub-5277424d-q5tk9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:49.008253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.008169 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw"] Oct 29 22:30:49.013396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.013378 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.026715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.026685 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:49.033061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.033037 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw"] Oct 29 22:30:49.051891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.051854 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5rx\" (UniqueName: \"kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.052182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.052150 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.052288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.052222 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.110077 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.110040 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j"] Oct 29 22:30:49.115740 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:49.115716 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103bf5a5_c7de_452c_9c7f_90f83e12f0bf.slice/crio-f746f16ea2bf4757854e7f620f2eb84ed5384a6f5c010a0b79993e3179e2757b WatchSource:0}: Error finding container f746f16ea2bf4757854e7f620f2eb84ed5384a6f5c010a0b79993e3179e2757b: Status 404 returned error can't find the container with id f746f16ea2bf4757854e7f620f2eb84ed5384a6f5c010a0b79993e3179e2757b Oct 29 22:30:49.118219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.118185 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:49.118304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.118252 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:49.118304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.118281 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:49.153019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.152992 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.153199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.153041 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.153199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.153102 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5rx\" (UniqueName: \"kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.153717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.153662 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.156169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.156142 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.177001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.176981 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5rx\" (UniqueName: \"kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx\") pod \"maas-api-revoke-sub-729c530e-b9mmw\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.327268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.327232 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:49.332945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.332921 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9"] Oct 29 22:30:49.337682 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:49.337655 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fd40b7_c6d7_408b_8718_cdf84853cdd9.slice/crio-cc6b1ef5b95a7f0b745aaa9c0175b9032f7bd46d675b02c3552ccf01523bfed1 WatchSource:0}: Error finding container cc6b1ef5b95a7f0b745aaa9c0175b9032f7bd46d675b02c3552ccf01523bfed1: Status 404 returned error can't find the container with id cc6b1ef5b95a7f0b745aaa9c0175b9032f7bd46d675b02c3552ccf01523bfed1 Oct 29 22:30:49.648665 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:49.648630 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129e0971_b687_4d26_ac9e_5615d099cc2e.slice/crio-a4b993c61f8afc74ff2c79115402b8de5a55290dd9227077445cc34f3340bc54 WatchSource:0}: Error finding container a4b993c61f8afc74ff2c79115402b8de5a55290dd9227077445cc34f3340bc54: Status 404 returned error can't find the container with id a4b993c61f8afc74ff2c79115402b8de5a55290dd9227077445cc34f3340bc54 Oct 29 22:30:49.648994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.648671 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw"] Oct 29 22:30:49.899397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.899315 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" event={"ID":"129e0971-b687-4d26-ac9e-5615d099cc2e","Type":"ContainerStarted","Data":"a4b993c61f8afc74ff2c79115402b8de5a55290dd9227077445cc34f3340bc54"} Oct 29 22:30:49.901918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.901893 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" event={"ID":"103bf5a5-c7de-452c-9c7f-90f83e12f0bf","Type":"ContainerStarted","Data":"c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1"} Oct 29 22:30:49.902048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.901946 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" event={"ID":"103bf5a5-c7de-452c-9c7f-90f83e12f0bf","Type":"ContainerStarted","Data":"f746f16ea2bf4757854e7f620f2eb84ed5384a6f5c010a0b79993e3179e2757b"} Oct 29 22:30:49.903655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.903627 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" event={"ID":"14fd40b7-c6d7-408b-8718-cdf84853cdd9","Type":"ContainerStarted","Data":"192a8ecc94ba5de2b953eacb5d07cf1a021b238bd61d3e2e909008a8f9e182da"} Oct 29 22:30:49.903752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.903672 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" event={"ID":"14fd40b7-c6d7-408b-8718-cdf84853cdd9","Type":"ContainerStarted","Data":"cc6b1ef5b95a7f0b745aaa9c0175b9032f7bd46d675b02c3552ccf01523bfed1"} Oct 29 22:30:49.953558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:49.952788 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podStartSLOduration=1.952769636 podStartE2EDuration="1.952769636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:49.951808043 +0000 UTC m=+916.845882964" watchObservedRunningTime="2026-10-29 22:30:49.952769636 +0000 UTC m=+916.846844558" Oct 29 22:30:50.039240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.039179 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" podStartSLOduration=2.039160985 podStartE2EDuration="2.039160985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:50.002997499 +0000 UTC m=+916.897072418" watchObservedRunningTime="2026-10-29 22:30:50.039160985 +0000 UTC m=+916.933235906" Oct 29 22:30:50.042749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.042724 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g"] Oct 29 22:30:50.046854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.046837 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.083721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.079198 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g"] Oct 29 22:30:50.101772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.092432 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.101772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.092617 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.101772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.092740 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xq7s\" (UniqueName: \"kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.193658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.193565 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.193658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.193641 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.193882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.193670 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xq7s\" (UniqueName: \"kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.194267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.194246 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.196049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.196028 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.213911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.213881 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xq7s\" (UniqueName: \"kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s\") pod \"maas-api-revoke-sub-2ee47d56-kl98g\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.387107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.387073 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:50.688397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.688367 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g"] Oct 29 22:30:50.689393 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:50.689369 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8a9438_9aba_4ea9_badd_f31bf8b77115.slice/crio-c98efed7ef9e5686e6544cb10c4eb1a6840eaccef5098121685cba8dd007fec3 WatchSource:0}: Error finding container c98efed7ef9e5686e6544cb10c4eb1a6840eaccef5098121685cba8dd007fec3: Status 404 returned error can't find the container with id c98efed7ef9e5686e6544cb10c4eb1a6840eaccef5098121685cba8dd007fec3 Oct 29 22:30:50.779894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.777873 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:50.780352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.780013 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:30:50.780352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.780063 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 22:30:50.911848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.911821 5367 generic.go:356] "Generic (PLEG): container finished" podID="129e0971-b687-4d26-ac9e-5615d099cc2e" containerID="30ee3a7973bc3560939a146fb449430c998d77f6322f326102a2aa6ba7a9e84f" exitCode=0 Oct 29 22:30:50.911976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.911858 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" event={"ID":"129e0971-b687-4d26-ac9e-5615d099cc2e","Type":"ContainerDied","Data":"30ee3a7973bc3560939a146fb449430c998d77f6322f326102a2aa6ba7a9e84f"} Oct 29 22:30:50.913277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.913254 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" event={"ID":"fe8a9438-9aba-4ea9-badd-f31bf8b77115","Type":"ContainerStarted","Data":"c98efed7ef9e5686e6544cb10c4eb1a6840eaccef5098121685cba8dd007fec3"} Oct 29 22:30:50.914694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.914674 5367 generic.go:356] "Generic (PLEG): container finished" podID="14fd40b7-c6d7-408b-8718-cdf84853cdd9" containerID="192a8ecc94ba5de2b953eacb5d07cf1a021b238bd61d3e2e909008a8f9e182da" exitCode=0 Oct 29 22:30:50.914809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:50.914725 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" event={"ID":"14fd40b7-c6d7-408b-8718-cdf84853cdd9","Type":"ContainerDied","Data":"192a8ecc94ba5de2b953eacb5d07cf1a021b238bd61d3e2e909008a8f9e182da"} Oct 29 22:30:51.775436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:51.775397 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:30:51.776052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:51.776020 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 22:30:51.923224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:51.923195 5367 generic.go:356] "Generic (PLEG): container finished" podID="fe8a9438-9aba-4ea9-badd-f31bf8b77115" containerID="b0fa4ed0cdf4fe4fdc06212da4921b8adb501fb243e5b5ca48c08a1c08630f74" exitCode=0 Oct 29 22:30:51.923355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:51.923277 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" event={"ID":"fe8a9438-9aba-4ea9-badd-f31bf8b77115","Type":"ContainerDied","Data":"b0fa4ed0cdf4fe4fdc06212da4921b8adb501fb243e5b5ca48c08a1c08630f74"} Oct 29 22:30:52.135786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.135758 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:52.190754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.190729 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:52.221824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.221789 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx\" (UniqueName: \"kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx\") pod \"129e0971-b687-4d26-ac9e-5615d099cc2e\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " Oct 29 22:30:52.222192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.221934 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca\") pod \"129e0971-b687-4d26-ac9e-5615d099cc2e\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " Oct 29 22:30:52.222344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.222318 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca\") pod \"129e0971-b687-4d26-ac9e-5615d099cc2e\" (UID: \"129e0971-b687-4d26-ac9e-5615d099cc2e\") " Oct 29 22:30:52.222489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.222394 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca\") pod \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " Oct 29 22:30:52.222489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.222444 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx\" (UniqueName: \"kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx\") pod \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " Oct 29 22:30:52.222638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.222502 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca\") pod \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\" (UID: \"14fd40b7-c6d7-408b-8718-cdf84853cdd9\") " Oct 29 22:30:52.224042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.224007 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca" pod "129e0971-b687-4d26-ac9e-5615d099cc2e" (UID: "129e0971-b687-4d26-ac9e-5615d099cc2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:52.224521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.224426 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx" pod "129e0971-b687-4d26-ac9e-5615d099cc2e" (UID: "129e0971-b687-4d26-ac9e-5615d099cc2e"). InnerVolumeSpecName "kube-api-access-ch5rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:52.224521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.224490 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca" pod "14fd40b7-c6d7-408b-8718-cdf84853cdd9" (UID: "14fd40b7-c6d7-408b-8718-cdf84853cdd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:52.225406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.225384 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca" pod "129e0971-b687-4d26-ac9e-5615d099cc2e" (UID: "129e0971-b687-4d26-ac9e-5615d099cc2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:52.225539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.225517 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca" pod "14fd40b7-c6d7-408b-8718-cdf84853cdd9" (UID: "14fd40b7-c6d7-408b-8718-cdf84853cdd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:52.226947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.226924 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx" pod "14fd40b7-c6d7-408b-8718-cdf84853cdd9" (UID: "14fd40b7-c6d7-408b-8718-cdf84853cdd9"). InnerVolumeSpecName "kube-api-access-hqjnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:52.327838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.326356 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.327838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.326382 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjnx\" (UniqueName: \"kubernetes.io/projected/14fd40b7-c6d7-408b-8718-cdf84853cdd9-kube-api-access-hqjnx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.327838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.326391 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14fd40b7-c6d7-408b-8718-cdf84853cdd9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.327838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.326400 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5rx\" (UniqueName: \"kubernetes.io/projected/129e0971-b687-4d26-ac9e-5615d099cc2e-kube-api-access-ch5rx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.327838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.326412 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.327838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.326423 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129e0971-b687-4d26-ac9e-5615d099cc2e-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.331601 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:30:52.331569 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:30:52.776910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.776813 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:30:52.777486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.777452 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 22:30:52.932279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.932253 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" Oct 29 22:30:52.932452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.932291 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw" event={"ID":"129e0971-b687-4d26-ac9e-5615d099cc2e","Type":"ContainerDied","Data":"a4b993c61f8afc74ff2c79115402b8de5a55290dd9227077445cc34f3340bc54"} Oct 29 22:30:52.932452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.932358 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b993c61f8afc74ff2c79115402b8de5a55290dd9227077445cc34f3340bc54" Oct 29 22:30:52.933754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.933729 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" event={"ID":"14fd40b7-c6d7-408b-8718-cdf84853cdd9","Type":"ContainerDied","Data":"cc6b1ef5b95a7f0b745aaa9c0175b9032f7bd46d675b02c3552ccf01523bfed1"} Oct 29 22:30:52.933754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.933747 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9" Oct 29 22:30:52.933929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:52.933774 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6b1ef5b95a7f0b745aaa9c0175b9032f7bd46d675b02c3552ccf01523bfed1" Oct 29 22:30:53.129766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.129728 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:53.237405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.237380 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca\") pod \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " Oct 29 22:30:53.237576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.237495 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca\") pod \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " Oct 29 22:30:53.237576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.237556 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s\" (UniqueName: \"kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s\") pod \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\" (UID: \"fe8a9438-9aba-4ea9-badd-f31bf8b77115\") " Oct 29 22:30:53.237864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.237842 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca" pod "fe8a9438-9aba-4ea9-badd-f31bf8b77115" (UID: "fe8a9438-9aba-4ea9-badd-f31bf8b77115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:53.238044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.238022 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:53.240039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.240015 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca" pod "fe8a9438-9aba-4ea9-badd-f31bf8b77115" (UID: "fe8a9438-9aba-4ea9-badd-f31bf8b77115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:53.240122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.240056 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s" pod "fe8a9438-9aba-4ea9-badd-f31bf8b77115" (UID: "fe8a9438-9aba-4ea9-badd-f31bf8b77115"). InnerVolumeSpecName "kube-api-access-4xq7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:53.339460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.339430 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xq7s\" (UniqueName: \"kubernetes.io/projected/fe8a9438-9aba-4ea9-badd-f31bf8b77115-kube-api-access-4xq7s\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:53.339460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.339461 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8a9438-9aba-4ea9-badd-f31bf8b77115-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:30:53.774114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.774076 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:30:53.774398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.774371 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 22:30:53.943085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.943050 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" event={"ID":"fe8a9438-9aba-4ea9-badd-f31bf8b77115","Type":"ContainerDied","Data":"c98efed7ef9e5686e6544cb10c4eb1a6840eaccef5098121685cba8dd007fec3"} Oct 29 22:30:53.943554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.943101 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c98efed7ef9e5686e6544cb10c4eb1a6840eaccef5098121685cba8dd007fec3" Oct 29 22:30:53.943554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:53.943104 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g" Oct 29 22:30:54.778876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:54.778833 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:30:54.779062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:54.778895 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 22:30:55.774820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:55.774790 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:55.775244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:55.775231 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:55.776091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:55.776072 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:30:58.981075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981036 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz"] Oct 29 22:30:58.981567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981554 5367 cpu_manager.go:414] "containerMap: removing container" podUID="129e0971-b687-4d26-ac9e-5615d099cc2e" containerName="revoke-keys" Oct 29 22:30:58.981605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981568 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129e0971-b687-4d26-ac9e-5615d099cc2e" containerName="revoke-keys" Oct 29 22:30:58.981605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981600 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fe8a9438-9aba-4ea9-badd-f31bf8b77115" containerName="revoke-keys" Oct 29 22:30:58.981605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981605 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8a9438-9aba-4ea9-badd-f31bf8b77115" containerName="revoke-keys" Oct 29 22:30:58.981716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981615 5367 cpu_manager.go:414] "containerMap: removing container" podUID="14fd40b7-c6d7-408b-8718-cdf84853cdd9" containerName="revoke-keys" Oct 29 22:30:58.981716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981621 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fd40b7-c6d7-408b-8718-cdf84853cdd9" containerName="revoke-keys" Oct 29 22:30:58.981716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981715 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="129e0971-b687-4d26-ac9e-5615d099cc2e" containerName="revoke-keys" Oct 29 22:30:58.981813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981730 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8a9438-9aba-4ea9-badd-f31bf8b77115" containerName="revoke-keys" Oct 29 22:30:58.981813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.981736 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fd40b7-c6d7-408b-8718-cdf84853cdd9" containerName="revoke-keys" Oct 29 22:30:58.985128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:58.985108 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.006135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.006108 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz"] Oct 29 22:30:59.013895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.013868 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:59.014063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.014044 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:30:59.014145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.014102 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:30:59.014211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.014159 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:30:59.053068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.053035 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.053252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.053165 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwjb\" (UniqueName: \"kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.053313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.053278 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.154259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.154225 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.154438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.154384 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.154501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.154489 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwjb\" (UniqueName: \"kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.154981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.154959 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.156784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.156758 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.174918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.174896 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwjb\" (UniqueName: \"kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb\") pod \"maas-api-revoke-sub-647a4e3f-gk7bz\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.301867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.301793 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:30:59.600377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:30:59.600347 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz"] Oct 29 22:30:59.601035 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:30:59.601007 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52566829_6e5d_4a4f_8777_ce55ab488111.slice/crio-98ecdd8d7fc3255714f6e9aa4af8461e6fd2431a235980e6d8007c250df56a35 WatchSource:0}: Error finding container 98ecdd8d7fc3255714f6e9aa4af8461e6fd2431a235980e6d8007c250df56a35: Status 404 returned error can't find the container with id 98ecdd8d7fc3255714f6e9aa4af8461e6fd2431a235980e6d8007c250df56a35 Oct 29 22:31:00.027013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:00.026971 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" event={"ID":"52566829-6e5d-4a4f-8777-ce55ab488111","Type":"ContainerStarted","Data":"226e4ab0bfcd9c59ba396c58ce68f31124e118ec39670dedc6292ef60b97b3fe"} Oct 29 22:31:00.027494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:00.027066 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" event={"ID":"52566829-6e5d-4a4f-8777-ce55ab488111","Type":"ContainerStarted","Data":"98ecdd8d7fc3255714f6e9aa4af8461e6fd2431a235980e6d8007c250df56a35"} Oct 29 22:31:00.072590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:00.072538 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" podStartSLOduration=2.072525529 podStartE2EDuration="2.072525529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:00.066826099 +0000 UTC m=+926.960901020" watchObservedRunningTime="2026-10-29 22:31:00.072525529 +0000 UTC m=+926.966600511" Oct 29 22:31:01.036228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.036195 5367 generic.go:356] "Generic (PLEG): container finished" podID="52566829-6e5d-4a4f-8777-ce55ab488111" containerID="226e4ab0bfcd9c59ba396c58ce68f31124e118ec39670dedc6292ef60b97b3fe" exitCode=0 Oct 29 22:31:01.036659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.036280 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" event={"ID":"52566829-6e5d-4a4f-8777-ce55ab488111","Type":"ContainerDied","Data":"226e4ab0bfcd9c59ba396c58ce68f31124e118ec39670dedc6292ef60b97b3fe"} Oct 29 22:31:01.887955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.887915 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p"] Oct 29 22:31:01.892126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.892102 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:01.922942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.922913 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"distinct-2-w1-879596-kserve-self-signed-certs\"" Oct 29 22:31:01.929395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.929365 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p"] Oct 29 22:31:01.989748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.989033 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:01.989748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.989144 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:01.989748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.989257 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:01.989748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.989316 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcss4\" (UniqueName: \"kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:01.989748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.989367 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:01.989748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:01.989413 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.090503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.090436 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.091048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.090899 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.104817 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.104993 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.105060 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcss4\" (UniqueName: \"kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.105113 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.105167 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.105584 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.117343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.110184 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.137347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.127098 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.137347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.133510 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcss4\" (UniqueName: \"kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.137347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.137286 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs\") pod \"distinct-2-w1-879596-kserve-b747c7575-7gq2p\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.205294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.205253 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:02.261849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.261819 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:31:02.398930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.398885 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4"] Oct 29 22:31:02.399455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.399440 5367 cpu_manager.go:414] "containerMap: removing container" podUID="52566829-6e5d-4a4f-8777-ce55ab488111" containerName="revoke-keys" Oct 29 22:31:02.399455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.399454 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52566829-6e5d-4a4f-8777-ce55ab488111" containerName="revoke-keys" Oct 29 22:31:02.399616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.399571 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="52566829-6e5d-4a4f-8777-ce55ab488111" containerName="revoke-keys" Oct 29 22:31:02.403107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.403085 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.408447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.408392 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca\") pod \"52566829-6e5d-4a4f-8777-ce55ab488111\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " Oct 29 22:31:02.408447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.408442 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca\") pod \"52566829-6e5d-4a4f-8777-ce55ab488111\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " Oct 29 22:31:02.408604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.408560 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb\" (UniqueName: \"kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb\") pod \"52566829-6e5d-4a4f-8777-ce55ab488111\" (UID: \"52566829-6e5d-4a4f-8777-ce55ab488111\") " Oct 29 22:31:02.408885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.408859 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca" pod "52566829-6e5d-4a4f-8777-ce55ab488111" (UID: "52566829-6e5d-4a4f-8777-ce55ab488111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:02.409079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.409062 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.411066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.411041 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca" pod "52566829-6e5d-4a4f-8777-ce55ab488111" (UID: "52566829-6e5d-4a4f-8777-ce55ab488111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:02.411151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.411123 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb" pod "52566829-6e5d-4a4f-8777-ce55ab488111" (UID: "52566829-6e5d-4a4f-8777-ce55ab488111"). InnerVolumeSpecName "kube-api-access-mlwjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:02.424165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.424138 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4"] Oct 29 22:31:02.510146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.510103 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.510325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.510252 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.510377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.510361 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcb9\" (UniqueName: \"kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.510442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.510423 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwjb\" (UniqueName: \"kubernetes.io/projected/52566829-6e5d-4a4f-8777-ce55ab488111-kube-api-access-mlwjb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.510502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.510446 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52566829-6e5d-4a4f-8777-ce55ab488111-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.529204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.529178 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p"] Oct 29 22:31:02.529406 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:02.529385 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5c9e995_c1bf_4f07_ac73_73248472db5d.slice/crio-3d82f24eea1840939c09ecee70c1b2df5d98349c0b4e2b6226b34ac598ff6064 WatchSource:0}: Error finding container 3d82f24eea1840939c09ecee70c1b2df5d98349c0b4e2b6226b34ac598ff6064: Status 404 returned error can't find the container with id 3d82f24eea1840939c09ecee70c1b2df5d98349c0b4e2b6226b34ac598ff6064 Oct 29 22:31:02.611302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.611256 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.611478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.611373 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcb9\" (UniqueName: \"kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.611545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.611493 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.612149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.612124 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.613798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.613775 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.634148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.634122 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcb9\" (UniqueName: \"kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9\") pod \"maas-api-revoke-sub-ff38a7d0-8kxh4\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:02.720311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:02.720236 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:03.020862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.020830 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4"] Oct 29 22:31:03.020960 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:03.020841 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791d6ae9_d240_49e3_aa64_5b71f7287493.slice/crio-d262c766c2d8a5ded4453ea8d197d8a89022f22813c0a63e29af85093d46116d WatchSource:0}: Error finding container d262c766c2d8a5ded4453ea8d197d8a89022f22813c0a63e29af85093d46116d: Status 404 returned error can't find the container with id d262c766c2d8a5ded4453ea8d197d8a89022f22813c0a63e29af85093d46116d Oct 29 22:31:03.051345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.051307 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" event={"ID":"52566829-6e5d-4a4f-8777-ce55ab488111","Type":"ContainerDied","Data":"98ecdd8d7fc3255714f6e9aa4af8461e6fd2431a235980e6d8007c250df56a35"} Oct 29 22:31:03.051471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.051366 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ecdd8d7fc3255714f6e9aa4af8461e6fd2431a235980e6d8007c250df56a35" Oct 29 22:31:03.051471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.051319 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz" Oct 29 22:31:03.053118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.053091 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" event={"ID":"b5c9e995-c1bf-4f07-ac73-73248472db5d","Type":"ContainerStarted","Data":"f154416c30e87d697094fc766ca80b00e84b1ff088ac105ea06b5fa92b376194"} Oct 29 22:31:03.053203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.053132 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" event={"ID":"b5c9e995-c1bf-4f07-ac73-73248472db5d","Type":"ContainerStarted","Data":"3d82f24eea1840939c09ecee70c1b2df5d98349c0b4e2b6226b34ac598ff6064"} Oct 29 22:31:03.054446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.054417 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" event={"ID":"791d6ae9-d240-49e3-aa64-5b71f7287493","Type":"ContainerStarted","Data":"d262c766c2d8a5ded4453ea8d197d8a89022f22813c0a63e29af85093d46116d"} Oct 29 22:31:03.098352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.098305 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podStartSLOduration=2.098292437 podStartE2EDuration="2.098292437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:03.095881295 +0000 UTC m=+929.989956216" watchObservedRunningTime="2026-10-29 22:31:03.098292437 +0000 UTC m=+929.992367412" Oct 29 22:31:03.206194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:03.206166 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:04.062996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:04.062964 5367 generic.go:356] "Generic (PLEG): container finished" podID="791d6ae9-d240-49e3-aa64-5b71f7287493" containerID="3a61a110c56a2c94b8d881a184c54c504e61666d33788f7257b5d300c8dbaa82" exitCode=0 Oct 29 22:31:04.063144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:04.063051 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" event={"ID":"791d6ae9-d240-49e3-aa64-5b71f7287493","Type":"ContainerDied","Data":"3a61a110c56a2c94b8d881a184c54c504e61666d33788f7257b5d300c8dbaa82"} Oct 29 22:31:05.230568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.230543 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:05.347379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.347305 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca\") pod \"791d6ae9-d240-49e3-aa64-5b71f7287493\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " Oct 29 22:31:05.347538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.347386 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9\" (UniqueName: \"kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9\") pod \"791d6ae9-d240-49e3-aa64-5b71f7287493\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " Oct 29 22:31:05.347538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.347528 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca\") pod \"791d6ae9-d240-49e3-aa64-5b71f7287493\" (UID: \"791d6ae9-d240-49e3-aa64-5b71f7287493\") " Oct 29 22:31:05.347920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.347890 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca" pod "791d6ae9-d240-49e3-aa64-5b71f7287493" (UID: "791d6ae9-d240-49e3-aa64-5b71f7287493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:05.348074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.348054 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:05.349373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.349350 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca" pod "791d6ae9-d240-49e3-aa64-5b71f7287493" (UID: "791d6ae9-d240-49e3-aa64-5b71f7287493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:05.349522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.349500 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9" pod "791d6ae9-d240-49e3-aa64-5b71f7287493" (UID: "791d6ae9-d240-49e3-aa64-5b71f7287493"). InnerVolumeSpecName "kube-api-access-zfcb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:05.449733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.449688 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d6ae9-d240-49e3-aa64-5b71f7287493-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:05.449733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:05.449729 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcb9\" (UniqueName: \"kubernetes.io/projected/791d6ae9-d240-49e3-aa64-5b71f7287493-kube-api-access-zfcb9\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:06.078358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:06.078321 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" event={"ID":"791d6ae9-d240-49e3-aa64-5b71f7287493","Type":"ContainerDied","Data":"d262c766c2d8a5ded4453ea8d197d8a89022f22813c0a63e29af85093d46116d"} Oct 29 22:31:06.078568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:06.078375 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d262c766c2d8a5ded4453ea8d197d8a89022f22813c0a63e29af85093d46116d" Oct 29 22:31:06.078568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:06.078342 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4" Oct 29 22:31:12.209722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:12.209166 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:12.761727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:12.760841 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:12.936721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:12.935808 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:31:12.999330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:12.999293 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:13.008727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:13.005003 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:13.008727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:13.005431 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:31:13.025881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:13.025780 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:13.092867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:13.092800 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:13.204723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:13.203824 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:31:18.124282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.124246 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs"] Oct 29 22:31:18.124828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.124812 5367 cpu_manager.go:414] "containerMap: removing container" podUID="791d6ae9-d240-49e3-aa64-5b71f7287493" containerName="revoke-keys" Oct 29 22:31:18.124828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.124828 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791d6ae9-d240-49e3-aa64-5b71f7287493" containerName="revoke-keys" Oct 29 22:31:18.124944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.124934 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="791d6ae9-d240-49e3-aa64-5b71f7287493" containerName="revoke-keys" Oct 29 22:31:18.129621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.129603 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.147942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.147915 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnk6\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.147951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148061 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148102 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148188 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148293 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148332 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148425 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148409 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.148485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.148471 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.167231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.167199 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs"] Oct 29 22:31:18.249569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.249537 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.249796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.249645 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.249796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.249684 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.249796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.249756 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.249897 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.249944 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250084 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250175 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250181 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250272 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnk6\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250424 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250177 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250554 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250818 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.250908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.250883 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.252510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.252482 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.252616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.252537 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.270436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.270400 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.270622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.270601 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnk6\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.443123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.443046 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:18.674937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.674901 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb"] Oct 29 22:31:18.680206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.680184 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.706204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.706134 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb"] Oct 29 22:31:18.756959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.756933 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757000 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757138 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757182 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxj2\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757233 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757259 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757313 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757376 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.757553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.757523 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.758233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.758216 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs"] Oct 29 22:31:18.758870 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:18.758848 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cd45f2_b51b_4edb_a5eb_3c66a9a5193f.slice/crio-0216b87face5d4550610d3245f64fc9d38a6fb29b14db46c31980f2592daa3c0 WatchSource:0}: Error finding container 0216b87face5d4550610d3245f64fc9d38a6fb29b14db46c31980f2592daa3c0: Status 404 returned error can't find the container with id 0216b87face5d4550610d3245f64fc9d38a6fb29b14db46c31980f2592daa3c0 Oct 29 22:31:18.761137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.761104 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:18.761216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.761165 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:18.761216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.761200 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:18.858928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.858891 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.859094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.858947 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.859094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.858983 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.859231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859100 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.859231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859220 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859380 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859470 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859514 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859524 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859728 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxj2\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859867 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859888 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859903 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.860080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.859973 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.862060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.862040 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.862634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.862614 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.879305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.879273 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.879502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.879477 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxj2\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:18.995126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:18.995039 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:19.235549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.235508 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" event={"ID":"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f","Type":"ContainerStarted","Data":"359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0"} Oct 29 22:31:19.236002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.235564 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" event={"ID":"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f","Type":"ContainerStarted","Data":"0216b87face5d4550610d3245f64fc9d38a6fb29b14db46c31980f2592daa3c0"} Oct 29 22:31:19.288384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.288252 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podStartSLOduration=1.2882397239999999 podStartE2EDuration="1.288239724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:19.284997911 +0000 UTC m=+946.179072831" watchObservedRunningTime="2026-10-29 22:31:19.288239724 +0000 UTC m=+946.182314643" Oct 29 22:31:19.332830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.332802 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb"] Oct 29 22:31:19.335392 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:19.335362 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31010b6_2721_4990_a044_68d8c9f0ccaf.slice/crio-d1c77592d6b707532ece3dd257852dd778d23654318e0d9cc6bd4dd1447e893f WatchSource:0}: Error finding container d1c77592d6b707532ece3dd257852dd778d23654318e0d9cc6bd4dd1447e893f: Status 404 returned error can't find the container with id d1c77592d6b707532ece3dd257852dd778d23654318e0d9cc6bd4dd1447e893f Oct 29 22:31:19.337965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.337928 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:19.338072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.337995 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:19.338072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.338024 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:19.443798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.443774 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:19.445473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.445448 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:19.445587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:19.445497 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:20.245179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:20.245140 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" event={"ID":"f31010b6-2721-4990-a044-68d8c9f0ccaf","Type":"ContainerStarted","Data":"4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35"} Oct 29 22:31:20.245567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:20.245204 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" event={"ID":"f31010b6-2721-4990-a044-68d8c9f0ccaf","Type":"ContainerStarted","Data":"d1c77592d6b707532ece3dd257852dd778d23654318e0d9cc6bd4dd1447e893f"} Oct 29 22:31:20.315263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:20.315210 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podStartSLOduration=2.315197664 podStartE2EDuration="2.315197664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:20.314302937 +0000 UTC m=+947.208377858" watchObservedRunningTime="2026-10-29 22:31:20.315197664 +0000 UTC m=+947.209272585" Oct 29 22:31:20.443946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:20.443909 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:20.444121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:20.443965 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:21.009747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.006208 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:21.009747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.007188 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.009747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.007236 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 29 22:31:21.443604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.443569 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.444044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.443629 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:21.508765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.499138 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw"] Oct 29 22:31:21.508765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.508121 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.513291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.513259 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw"] Oct 29 22:31:21.524842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.524791 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:31:21.525046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.524947 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:31:21.525093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.525011 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:31:21.525093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.525059 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:31:21.621247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.621210 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpsc\" (UniqueName: \"kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.621425 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.621354 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.621494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.621474 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.722395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.722310 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpsc\" (UniqueName: \"kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.722577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.722423 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.722577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.722491 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.723674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.723626 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.724855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.724836 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.742647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.742619 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpsc\" (UniqueName: \"kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc\") pod \"maas-api-revoke-sub-50b857fc-wp9bw\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:21.823963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:21.823932 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:22.015487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.011795 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.015487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.011850 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 29 22:31:22.210195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.210117 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw"] Oct 29 22:31:22.340227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.340187 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" event={"ID":"b251e382-ed22-417c-a854-2e20d97f35c4","Type":"ContainerStarted","Data":"1c01902b6067e1ffdf0790a0a395d1cfb01d60b591f2bea046237796ae3851d0"} Oct 29 22:31:22.390296 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:31:22.390257 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:31:22.443864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.443831 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.444239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.443884 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:23.001780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:22.999997 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.001780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:23.000067 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 29 22:31:23.349501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:23.349465 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" event={"ID":"b251e382-ed22-417c-a854-2e20d97f35c4","Type":"ContainerStarted","Data":"50e46f0231c3f258e31d5073868ddcba61ae7e5a5261a4b40740e051546c50bd"} Oct 29 22:31:23.406723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:23.406047 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" podStartSLOduration=2.406032692 podStartE2EDuration="2.406032692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:23.390888869 +0000 UTC m=+950.284963791" watchObservedRunningTime="2026-10-29 22:31:23.406032692 +0000 UTC m=+950.300107612" Oct 29 22:31:23.443858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:23.443833 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.443967 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:23.443889 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:24.002722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.001853 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.002722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.001922 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 29 22:31:24.382741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.381885 5367 generic.go:356] "Generic (PLEG): container finished" podID="b251e382-ed22-417c-a854-2e20d97f35c4" containerID="50e46f0231c3f258e31d5073868ddcba61ae7e5a5261a4b40740e051546c50bd" exitCode=0 Oct 29 22:31:24.382741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.382183 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" event={"ID":"b251e382-ed22-417c-a854-2e20d97f35c4","Type":"ContainerDied","Data":"50e46f0231c3f258e31d5073868ddcba61ae7e5a5261a4b40740e051546c50bd"} Oct 29 22:31:24.444179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.444145 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.444555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.444201 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:24.995752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.995716 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.995925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:24.995777 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 29 22:31:25.443856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.443821 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.444027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.443897 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:31:25.591797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.591748 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:25.720627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.720526 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca\") pod \"b251e382-ed22-417c-a854-2e20d97f35c4\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " Oct 29 22:31:25.720819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.720732 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc\" (UniqueName: \"kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc\") pod \"b251e382-ed22-417c-a854-2e20d97f35c4\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " Oct 29 22:31:25.720819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.720765 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca\") pod \"b251e382-ed22-417c-a854-2e20d97f35c4\" (UID: \"b251e382-ed22-417c-a854-2e20d97f35c4\") " Oct 29 22:31:25.721014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.720984 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca" pod "b251e382-ed22-417c-a854-2e20d97f35c4" (UID: "b251e382-ed22-417c-a854-2e20d97f35c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:25.721382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.721355 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:25.722886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.722850 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca" pod "b251e382-ed22-417c-a854-2e20d97f35c4" (UID: "b251e382-ed22-417c-a854-2e20d97f35c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:25.723417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.723387 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc" pod "b251e382-ed22-417c-a854-2e20d97f35c4" (UID: "b251e382-ed22-417c-a854-2e20d97f35c4"). InnerVolumeSpecName "kube-api-access-svpsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:25.822986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.822930 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svpsc\" (UniqueName: \"kubernetes.io/projected/b251e382-ed22-417c-a854-2e20d97f35c4-kube-api-access-svpsc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:25.822986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.822987 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b251e382-ed22-417c-a854-2e20d97f35c4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:25.995844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.995753 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.995844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:25.995809 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 29 22:31:26.409344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:26.409309 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" Oct 29 22:31:26.409344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:26.409329 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw" event={"ID":"b251e382-ed22-417c-a854-2e20d97f35c4","Type":"ContainerDied","Data":"1c01902b6067e1ffdf0790a0a395d1cfb01d60b591f2bea046237796ae3851d0"} Oct 29 22:31:26.409568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:26.409407 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c01902b6067e1ffdf0790a0a395d1cfb01d60b591f2bea046237796ae3851d0" Oct 29 22:31:26.447798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:26.447764 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:26.448221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:26.448198 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:26.448947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:26.448926 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 22:31:27.001764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.001736 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:27.002185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.002175 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:27.003060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.003038 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:31:27.413857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.413823 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6"] Oct 29 22:31:27.414519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.414498 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b251e382-ed22-417c-a854-2e20d97f35c4" containerName="revoke-keys" Oct 29 22:31:27.414519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.414520 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b251e382-ed22-417c-a854-2e20d97f35c4" containerName="revoke-keys" Oct 29 22:31:27.414740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.414726 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b251e382-ed22-417c-a854-2e20d97f35c4" containerName="revoke-keys" Oct 29 22:31:27.418774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.418751 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.433565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.433540 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6"] Oct 29 22:31:27.434864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.434843 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"openshift-service-ca.crt\"" Oct 29 22:31:27.434994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.434852 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"default-dockercfg-f6qvf\"" Oct 29 22:31:27.434994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.434896 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"kube-root-ca.crt\"" Oct 29 22:31:27.442414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.442375 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"distinct-2-w6-92fd86-kserve-self-signed-certs\"" Oct 29 22:31:27.546337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.546308 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.546538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.546353 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.546629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.546607 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxxk\" (UniqueName: \"kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.547024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.547001 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.547202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.547183 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.547281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.547219 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.648566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.648525 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxxk\" (UniqueName: \"kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.648757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.648680 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.648812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.648794 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.648865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.648822 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.648913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.648892 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.648953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.648935 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.649164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.649141 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.649278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.649216 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.649326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.649280 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.651552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.651527 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.651878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.651857 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.681785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.680471 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxxk\" (UniqueName: \"kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk\") pod \"distinct-2-w6-92fd86-kserve-89d548559-4s9l6\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:27.735214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:27.735179 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:28.050920 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:28.050884 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd926f8d9_c648_4312_9df4_488ffa2b300a.slice/crio-5b265ac408634ea6df33b38388f22ece454f4c5ad6b467c2382a9ad7890885a5 WatchSource:0}: Error finding container 5b265ac408634ea6df33b38388f22ece454f4c5ad6b467c2382a9ad7890885a5: Status 404 returned error can't find the container with id 5b265ac408634ea6df33b38388f22ece454f4c5ad6b467c2382a9ad7890885a5 Oct 29 22:31:28.050920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:28.050891 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6"] Oct 29 22:31:28.427760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:28.427690 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" event={"ID":"d926f8d9-c648-4312-9df4-488ffa2b300a","Type":"ContainerStarted","Data":"428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4"} Oct 29 22:31:28.427965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:28.427775 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" event={"ID":"d926f8d9-c648-4312-9df4-488ffa2b300a","Type":"ContainerStarted","Data":"5b265ac408634ea6df33b38388f22ece454f4c5ad6b467c2382a9ad7890885a5"} Oct 29 22:31:28.469335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:28.469281 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podStartSLOduration=1.469266951 podStartE2EDuration="1.469266951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:28.468723158 +0000 UTC m=+955.362798075" watchObservedRunningTime="2026-10-29 22:31:28.469266951 +0000 UTC m=+955.363341887" Oct 29 22:31:28.735518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:28.735438 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:33.394772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.394743 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6"] Oct 29 22:31:33.400157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.400138 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.427205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.427179 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6"] Oct 29 22:31:33.552292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552251 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552307 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552444 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552484 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552558 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552624 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552725 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552797 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.552864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.552853 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654374 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654420 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654474 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654513 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654545 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654626 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654665 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654732 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654773 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.654923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654870 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.655079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654928 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.655079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.654996 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.655079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.655068 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.655402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.655378 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.656940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.656922 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.657028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.657006 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.693620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.693593 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.694173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.694151 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:33.713880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:33.713859 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:34.029852 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:34.029822 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a9f08_4c97_481b_88bd_7f48c6f0e488.slice/crio-b83e29310b5d8e900ec1eaae1a82ab98e1e01516f9297b4cb56dd8b2ab545514 WatchSource:0}: Error finding container b83e29310b5d8e900ec1eaae1a82ab98e1e01516f9297b4cb56dd8b2ab545514: Status 404 returned error can't find the container with id b83e29310b5d8e900ec1eaae1a82ab98e1e01516f9297b4cb56dd8b2ab545514 Oct 29 22:31:34.029963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.029944 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6"] Oct 29 22:31:34.032423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.032386 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:34.032536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.032461 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:34.032536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.032507 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:34.506056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.506017 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" event={"ID":"c15a9f08-4c97-481b-88bd-7f48c6f0e488","Type":"ContainerStarted","Data":"83c72c5c8a35ab076d6ca3919b5f26d62050923ba8a09fabe9f051d844ef62f3"} Oct 29 22:31:34.506501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.506081 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" event={"ID":"c15a9f08-4c97-481b-88bd-7f48c6f0e488","Type":"ContainerStarted","Data":"b83e29310b5d8e900ec1eaae1a82ab98e1e01516f9297b4cb56dd8b2ab545514"} Oct 29 22:31:34.583097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.583040 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podStartSLOduration=1.58302613 podStartE2EDuration="1.58302613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:34.580157705 +0000 UTC m=+961.474232624" watchObservedRunningTime="2026-10-29 22:31:34.58302613 +0000 UTC m=+961.477101052" Oct 29 22:31:34.714860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:34.714829 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:35.717725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:35.717665 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:31:35.718163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:35.717775 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 29 22:31:36.715096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:36.715056 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:31:36.715270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:36.715151 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 29 22:31:37.714719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:37.714522 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:31:37.714719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:37.714577 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 29 22:31:37.735643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:37.735603 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:37.747778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:37.747751 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:38.553215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:38.552841 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:31:38.714446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:38.714395 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:31:38.714707 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:38.714669 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 29 22:31:39.714263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:39.714225 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:31:39.714815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:39.714289 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 29 22:31:40.362009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.361972 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq"] Oct 29 22:31:40.367850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.367812 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.389845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.389822 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq"] Oct 29 22:31:40.393742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.392886 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"distinct-w3-95672c-kserve-self-signed-certs\"" Oct 29 22:31:40.494227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.494192 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szn6\" (UniqueName: \"kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.494408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.494243 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.494408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.494272 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.494507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.494466 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.494613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.494601 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.494656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.494645 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.595558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.595526 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.595805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.595605 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.595805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.595715 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szn6\" (UniqueName: \"kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.595805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.595766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.595805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.595800 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.595982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.595862 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.596245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.596213 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.596344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.596277 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.596344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.596334 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.598913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.598886 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.599413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.599392 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.617316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.617240 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szn6\" (UniqueName: \"kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6\") pod \"distinct-w3-95672c-kserve-cc746d98d-k49xq\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.691403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.691368 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:40.720094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.720070 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:40.720816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.720757 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:40.721617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:40.721594 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 22:31:41.016147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:41.016070 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq"] Oct 29 22:31:41.024863 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:41.024826 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7a9c642_24bd_4b19_a233_0f635554b1dc.slice/crio-9d2e2f3251228f903bf9343784234e8d286f4cbe476728beec1b6e77c9ef2ef4 WatchSource:0}: Error finding container 9d2e2f3251228f903bf9343784234e8d286f4cbe476728beec1b6e77c9ef2ef4: Status 404 returned error can't find the container with id 9d2e2f3251228f903bf9343784234e8d286f4cbe476728beec1b6e77c9ef2ef4 Oct 29 22:31:41.573868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:41.573828 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" event={"ID":"c7a9c642-24bd-4b19-a233-0f635554b1dc","Type":"ContainerStarted","Data":"40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb"} Oct 29 22:31:41.574030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:41.573878 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" event={"ID":"c7a9c642-24bd-4b19-a233-0f635554b1dc","Type":"ContainerStarted","Data":"9d2e2f3251228f903bf9343784234e8d286f4cbe476728beec1b6e77c9ef2ef4"} Oct 29 22:31:41.619204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:41.619153 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podStartSLOduration=1.619136142 podStartE2EDuration="1.619136142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:41.615428932 +0000 UTC m=+968.509503853" watchObservedRunningTime="2026-10-29 22:31:41.619136142 +0000 UTC m=+968.513211082" Oct 29 22:31:41.691624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:41.691593 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:47.110724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:47.093856 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:31:47.201199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:47.199791 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:31:47.352724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:47.329916 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:48.252900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.252871 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt"] Oct 29 22:31:48.257402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.257379 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.267792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.267766 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb"] Oct 29 22:31:48.272110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.272091 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.284752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.284726 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb"] Oct 29 22:31:48.291838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.291811 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt"] Oct 29 22:31:48.295903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.295884 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:31:48.296136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.296107 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:31:48.296232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.296156 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:31:48.296279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.296233 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:31:48.333992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.333957 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.334166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334024 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qtq\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334172 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334217 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334257 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334289 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334377 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334420 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334520 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.334573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334562 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334625 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.334681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.334668 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knll8\" (UniqueName: \"kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.435388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435354 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435389 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435414 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435534 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435600 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435629 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435735 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.435838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435759 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435831 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.435948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435875 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.436045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.435945 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knll8\" (UniqueName: \"kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.436045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436032 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.436154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436095 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qtq\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.436154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436103 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.436154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436033 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.436335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436305 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.436623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436603 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.436843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.436826 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.437859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.437838 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.439811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.439786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.439899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.439877 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.455735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.455712 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.455990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.455970 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knll8\" (UniqueName: \"kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8\") pod \"maas-api-revoke-sub-51ed72aa-gkxdb\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.456031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.455970 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qtq\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.574159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.574109 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:48.587947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.587914 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:48.910956 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:48.910924 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263ffebb_1be0_4998_a15d_1b1477faf5ec.slice/crio-83320fb55fc7dca80a74b3d38a8020855568550f546a70a53d1c4023cbdb584a WatchSource:0}: Error finding container 83320fb55fc7dca80a74b3d38a8020855568550f546a70a53d1c4023cbdb584a: Status 404 returned error can't find the container with id 83320fb55fc7dca80a74b3d38a8020855568550f546a70a53d1c4023cbdb584a Oct 29 22:31:48.911084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.910985 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb"] Oct 29 22:31:48.913653 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:48.913622 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1635df12_4155_49bd_858b_3b4674a893c7.slice/crio-33c4d22cd29748900a91d61e2312fb88627c45fbc93006968e7a218768c5016e WatchSource:0}: Error finding container 33c4d22cd29748900a91d61e2312fb88627c45fbc93006968e7a218768c5016e: Status 404 returned error can't find the container with id 33c4d22cd29748900a91d61e2312fb88627c45fbc93006968e7a218768c5016e Oct 29 22:31:48.915942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.915922 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt"] Oct 29 22:31:48.916076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.916048 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:48.916125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.916112 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:48.916162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:48.916144 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:49.786773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:49.786733 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" event={"ID":"263ffebb-1be0-4998-a15d-1b1477faf5ec","Type":"ContainerStarted","Data":"5ba1a313a4405ff324bb885a95614138d35847e19569e103aea7612a5916d164"} Oct 29 22:31:49.787249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:49.786803 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" event={"ID":"263ffebb-1be0-4998-a15d-1b1477faf5ec","Type":"ContainerStarted","Data":"83320fb55fc7dca80a74b3d38a8020855568550f546a70a53d1c4023cbdb584a"} Oct 29 22:31:49.788383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:49.788358 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" event={"ID":"1635df12-4155-49bd-858b-3b4674a893c7","Type":"ContainerStarted","Data":"c6482d182bf5ce9a3fa93defa69b0e36d23a33a9ae8270e5f53a06950b797f30"} Oct 29 22:31:49.788514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:49.788401 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" event={"ID":"1635df12-4155-49bd-858b-3b4674a893c7","Type":"ContainerStarted","Data":"33c4d22cd29748900a91d61e2312fb88627c45fbc93006968e7a218768c5016e"} Oct 29 22:31:49.829070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:49.829028 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" podStartSLOduration=1.829015408 podStartE2EDuration="1.829015408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:49.827665849 +0000 UTC m=+976.721740766" watchObservedRunningTime="2026-10-29 22:31:49.829015408 +0000 UTC m=+976.723090328" Oct 29 22:31:49.883510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:49.883442 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podStartSLOduration=1.88342373 podStartE2EDuration="1.88342373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:49.879810328 +0000 UTC m=+976.773885247" watchObservedRunningTime="2026-10-29 22:31:49.88342373 +0000 UTC m=+976.777498654" Oct 29 22:31:50.578735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.576412 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:50.578944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.578828 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:31:50.578944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.578885 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:31:50.692056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.692017 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:50.705188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.705162 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:50.797566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.797533 5367 generic.go:356] "Generic (PLEG): container finished" podID="263ffebb-1be0-4998-a15d-1b1477faf5ec" containerID="5ba1a313a4405ff324bb885a95614138d35847e19569e103aea7612a5916d164" exitCode=0 Oct 29 22:31:50.797996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.797571 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" event={"ID":"263ffebb-1be0-4998-a15d-1b1477faf5ec","Type":"ContainerDied","Data":"5ba1a313a4405ff324bb885a95614138d35847e19569e103aea7612a5916d164"} Oct 29 22:31:50.809879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:50.809857 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:31:51.578724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:51.578107 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:31:51.578724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:51.578171 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:31:52.014547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.014518 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:52.123102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.123064 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca\") pod \"263ffebb-1be0-4998-a15d-1b1477faf5ec\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " Oct 29 22:31:52.123430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.123409 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca\") pod \"263ffebb-1be0-4998-a15d-1b1477faf5ec\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " Oct 29 22:31:52.123544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.123447 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8\" (UniqueName: \"kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8\") pod \"263ffebb-1be0-4998-a15d-1b1477faf5ec\" (UID: \"263ffebb-1be0-4998-a15d-1b1477faf5ec\") " Oct 29 22:31:52.125383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.125285 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca" pod "263ffebb-1be0-4998-a15d-1b1477faf5ec" (UID: "263ffebb-1be0-4998-a15d-1b1477faf5ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:52.127462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.127261 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca" pod "263ffebb-1be0-4998-a15d-1b1477faf5ec" (UID: "263ffebb-1be0-4998-a15d-1b1477faf5ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:52.127462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.127410 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8" pod "263ffebb-1be0-4998-a15d-1b1477faf5ec" (UID: "263ffebb-1be0-4998-a15d-1b1477faf5ec"). InnerVolumeSpecName "kube-api-access-knll8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:52.225002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.224962 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:52.225002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.224999 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263ffebb-1be0-4998-a15d-1b1477faf5ec-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:52.225002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.225012 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knll8\" (UniqueName: \"kubernetes.io/projected/263ffebb-1be0-4998-a15d-1b1477faf5ec-kube-api-access-knll8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:31:52.331381 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:31:52.331341 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:31:52.579002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.578839 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:31:52.579002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.578900 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:31:52.814118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.814082 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" Oct 29 22:31:52.814295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.814081 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb" event={"ID":"263ffebb-1be0-4998-a15d-1b1477faf5ec","Type":"ContainerDied","Data":"83320fb55fc7dca80a74b3d38a8020855568550f546a70a53d1c4023cbdb584a"} Oct 29 22:31:52.814295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:52.814211 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83320fb55fc7dca80a74b3d38a8020855568550f546a70a53d1c4023cbdb584a" Oct 29 22:31:53.578724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:53.576206 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:31:53.578724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:53.576267 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:31:54.332757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.329870 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7"] Oct 29 22:31:54.332757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.330856 5367 cpu_manager.go:414] "containerMap: removing container" podUID="263ffebb-1be0-4998-a15d-1b1477faf5ec" containerName="revoke-keys" Oct 29 22:31:54.332757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.330875 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263ffebb-1be0-4998-a15d-1b1477faf5ec" containerName="revoke-keys" Oct 29 22:31:54.332757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.331027 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="263ffebb-1be0-4998-a15d-1b1477faf5ec" containerName="revoke-keys" Oct 29 22:31:54.336420 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.336394 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.382475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.382412 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7"] Oct 29 22:31:54.382945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.382921 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"kube-root-ca.crt\"" Oct 29 22:31:54.383207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.383139 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"distinct-2-w5-0d12c5-kserve-self-signed-certs\"" Oct 29 22:31:54.383321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.383153 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"openshift-service-ca.crt\"" Oct 29 22:31:54.383427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.383203 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"default-dockercfg-9x74l\"" Oct 29 22:31:54.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.473534 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.473617 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.473661 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.473686 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljp5\" (UniqueName: \"kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.473862 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.475723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.473976 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.579634 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.579777 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.579826 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.579866 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.579890 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljp5\" (UniqueName: \"kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.580030 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.581071 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.581323 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.582722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.581618 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.583623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.583400 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.609186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.592403 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.609186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.592994 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:31:54.609186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.593044 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:31:54.618007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.615311 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljp5\" (UniqueName: \"kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5\") pod \"distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.651013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.650978 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:31:54.983886 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:31:54.983857 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7eea3c5_0fe1_42ce_876c_e44421cd3db1.slice/crio-79a2e13a323aa1162bb6e094d8c2e63854c5b77c0344c915aeeba7e0bc7c4d26 WatchSource:0}: Error finding container 79a2e13a323aa1162bb6e094d8c2e63854c5b77c0344c915aeeba7e0bc7c4d26: Status 404 returned error can't find the container with id 79a2e13a323aa1162bb6e094d8c2e63854c5b77c0344c915aeeba7e0bc7c4d26 Oct 29 22:31:54.984512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:54.984426 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7"] Oct 29 22:31:55.845573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:55.845531 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" event={"ID":"a7eea3c5-0fe1-42ce-876c-e44421cd3db1","Type":"ContainerStarted","Data":"499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7"} Oct 29 22:31:55.846101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:55.845584 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" event={"ID":"a7eea3c5-0fe1-42ce-876c-e44421cd3db1","Type":"ContainerStarted","Data":"79a2e13a323aa1162bb6e094d8c2e63854c5b77c0344c915aeeba7e0bc7c4d26"} Oct 29 22:31:55.866234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:55.866208 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:55.866679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:55.866650 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:55.868086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:55.868066 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:31:55.887478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:55.887431 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podStartSLOduration=1.887419238 podStartE2EDuration="1.887419238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:55.886132357 +0000 UTC m=+982.780207278" watchObservedRunningTime="2026-10-29 22:31:55.887419238 +0000 UTC m=+982.781494159" Oct 29 22:31:56.651742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:31:56.651688 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:32:02.718875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:02.718826 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:02.899724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:02.898860 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.045598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.043874 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:32:03.045598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.044899 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:32:03.046871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.043785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.092984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.092550 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.177982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.177900 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" podUID="fd4be442-0fd5-483a-a2fb-45561dfdaafc" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.208737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.199803 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:32:03.228769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.213285 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.283968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.282801 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.447725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.434794 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.671019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.669751 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt"] Oct 29 22:32:03.854719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.840947 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt"] Oct 29 22:32:03.854719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.841092 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.854719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.848301 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm"] Oct 29 22:32:03.886914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.886884 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9"] Oct 29 22:32:03.888686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.888662 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:03.933176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.933127 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:03.934792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.934759 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm"] Oct 29 22:32:03.943363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.941051 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.943363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.941104 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.943363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.941138 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.943363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.941163 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.943363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.941399 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.943363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.941673 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.943973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.943950 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.944110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.944094 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.944304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.944286 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjk6\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:03.946792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:03.946182 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9"] Oct 29 22:32:04.045460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045425 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.045643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045503 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.045643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045534 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.045643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045605 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.045643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045622 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.045907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045674 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.045907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045778 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.045907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045813 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.045907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045858 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.045907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.045885 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046001 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046028 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046081 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046123 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.046345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046238 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046276 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqnmp\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.046345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046293 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046360 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046381 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046406 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.046513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046435 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.046668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046542 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjk6\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046565 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046585 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046604 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.046668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046637 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2r6\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046672 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046719 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.046990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.046879 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.047129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.047110 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.047283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.047264 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.047400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.047385 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.049031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.049005 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.049185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.049167 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.067130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.067098 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.067528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.067503 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjk6\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.147854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147751 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2r6\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.147854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147795 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.147854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147816 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147880 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147911 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147957 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.147978 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148040 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148081 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148130 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148209 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148231 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148256 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148290 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148301 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148332 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148351 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148362 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148401 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqnmp\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148428 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148475 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148504 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148524 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.148610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148537 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.149388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.148642 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.149388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.149074 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.149388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.149282 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.149388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.149301 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.151079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.151059 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.151427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.151402 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.151741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.151721 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.151825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.151755 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.170252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.170169 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.172417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.171556 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqnmp\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.172417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.171847 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2r6\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.173008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.172981 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.249758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.249727 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:04.277091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.277064 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:04.292941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.292776 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:04.572626 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:04.572597 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3345ff1_2c5e_47c5_856a_2caac6909907.slice/crio-11a688d3df4eea86b9e92d214c76996095e8e9e994dc0d5b1613ff45f113b345 WatchSource:0}: Error finding container 11a688d3df4eea86b9e92d214c76996095e8e9e994dc0d5b1613ff45f113b345: Status 404 returned error can't find the container with id 11a688d3df4eea86b9e92d214c76996095e8e9e994dc0d5b1613ff45f113b345 Oct 29 22:32:04.572790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.572750 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt"] Oct 29 22:32:04.575190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.575159 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.575264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.575221 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.575264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.575251 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.610093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.610068 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm"] Oct 29 22:32:04.612945 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:04.612914 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168026fd_c56f_432c_ab83_85323cdda342.slice/crio-7f6b5fbe2fdddfa418b80cfcc9c6ade7b2453609cc142d67695434b99cb219de WatchSource:0}: Error finding container 7f6b5fbe2fdddfa418b80cfcc9c6ade7b2453609cc142d67695434b99cb219de: Status 404 returned error can't find the container with id 7f6b5fbe2fdddfa418b80cfcc9c6ade7b2453609cc142d67695434b99cb219de Oct 29 22:32:04.615490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.615457 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.615606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.615539 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.615606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.615586 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.630750 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:04.630687 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357cc0bd_e32e_48f4_9caf_f901888c530f.slice/crio-751a0c0192e1a0220fc9807bb733687e7efb0eb78320b63333c2478a2bd5eb06 WatchSource:0}: Error finding container 751a0c0192e1a0220fc9807bb733687e7efb0eb78320b63333c2478a2bd5eb06: Status 404 returned error can't find the container with id 751a0c0192e1a0220fc9807bb733687e7efb0eb78320b63333c2478a2bd5eb06 Oct 29 22:32:04.631539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.631514 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9"] Oct 29 22:32:04.634031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.633995 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.634148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.634070 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.634148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.634111 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:04.651754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.651155 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:32:04.671600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.671245 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:32:04.996769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.996427 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" event={"ID":"d3345ff1-2c5e-47c5-856a-2caac6909907","Type":"ContainerStarted","Data":"ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b"} Oct 29 22:32:04.996769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.996490 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" event={"ID":"d3345ff1-2c5e-47c5-856a-2caac6909907","Type":"ContainerStarted","Data":"11a688d3df4eea86b9e92d214c76996095e8e9e994dc0d5b1613ff45f113b345"} Oct 29 22:32:04.998132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.998101 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" event={"ID":"357cc0bd-e32e-48f4-9caf-f901888c530f","Type":"ContainerStarted","Data":"ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84"} Oct 29 22:32:04.998266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:04.998153 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" event={"ID":"357cc0bd-e32e-48f4-9caf-f901888c530f","Type":"ContainerStarted","Data":"751a0c0192e1a0220fc9807bb733687e7efb0eb78320b63333c2478a2bd5eb06"} Oct 29 22:32:05.002501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.002476 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" event={"ID":"168026fd-c56f-432c-ab83-85323cdda342","Type":"ContainerStarted","Data":"0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad"} Oct 29 22:32:05.003146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.003124 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" event={"ID":"168026fd-c56f-432c-ab83-85323cdda342","Type":"ContainerStarted","Data":"7f6b5fbe2fdddfa418b80cfcc9c6ade7b2453609cc142d67695434b99cb219de"} Oct 29 22:32:05.021446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.021415 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:32:05.048310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.048245 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podStartSLOduration=2.048226997 podStartE2EDuration="2.048226997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:05.047714827 +0000 UTC m=+991.941789743" watchObservedRunningTime="2026-10-29 22:32:05.048226997 +0000 UTC m=+991.942301917" Oct 29 22:32:05.106759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.106680 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podStartSLOduration=2.106658966 podStartE2EDuration="2.106658966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:05.091867272 +0000 UTC m=+991.985942192" watchObservedRunningTime="2026-10-29 22:32:05.106658966 +0000 UTC m=+992.000733888" Oct 29 22:32:05.138913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.138862 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podStartSLOduration=2.138847561 podStartE2EDuration="2.138847561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:05.13526817 +0000 UTC m=+992.029343094" watchObservedRunningTime="2026-10-29 22:32:05.138847561 +0000 UTC m=+992.032922481" Oct 29 22:32:05.250526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.250405 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:05.252050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.252023 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:05.252172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.252077 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:05.278056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.278030 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:05.280064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.280040 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:05.280168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.280089 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:05.293762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.293733 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:05.295594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.295569 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:05.295685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:05.295612 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:06.253723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:06.251083 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.253723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:06.251142 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:06.290722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:06.289947 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.290722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:06.290012 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:06.303724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:06.297052 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.303724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:06.297110 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:07.029724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.029382 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl"] Oct 29 22:32:07.035091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.035063 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.039801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.039195 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl"] Oct 29 22:32:07.140649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.140528 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:32:07.140876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.140735 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:07.142298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.141345 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:07.142298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.141415 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:32:07.142298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.141915 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.142298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.142189 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.142298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.142217 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjm9x\" (UniqueName: \"kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.244209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.244177 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.244445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.244432 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.244525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.244452 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm9x\" (UniqueName: \"kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.246066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.246046 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.248540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.248181 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.250344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.250324 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.250453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.250370 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:07.288773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.282076 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.288773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.282139 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:07.288773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.282446 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjm9x\" (UniqueName: \"kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x\") pod \"maas-api-revoke-sub-358c33ed-k2jnl\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:07.300726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.295063 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.300726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.295124 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:07.537032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:07.537000 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:08.338770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.336856 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.338770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.336926 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:08.338770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.337833 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.338770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.337875 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:08.343911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.343881 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.344091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.343929 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:08.534986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:08.534958 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl"] Oct 29 22:32:08.538273 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:08.538245 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9728a3_eefe_42db_bd84_e437b37ef405.slice/crio-ba2359e6517be942c1f7dc788a076d78220da4998a01bc9a631a78d9ca7bd6fc WatchSource:0}: Error finding container ba2359e6517be942c1f7dc788a076d78220da4998a01bc9a631a78d9ca7bd6fc: Status 404 returned error can't find the container with id ba2359e6517be942c1f7dc788a076d78220da4998a01bc9a631a78d9ca7bd6fc Oct 29 22:32:09.250399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.250363 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.250550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.250424 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:09.279793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.279090 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.279793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.279162 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:09.297725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.296756 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.297725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.296810 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:09.430935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.429524 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" event={"ID":"da9728a3-eefe-42db-bd84-e437b37ef405","Type":"ContainerStarted","Data":"50bfbfc8bcb0db32c61fce5f5d6ef5ad1da449dff7721c3042b2283d66889dab"} Oct 29 22:32:09.430935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:09.429598 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" event={"ID":"da9728a3-eefe-42db-bd84-e437b37ef405","Type":"ContainerStarted","Data":"ba2359e6517be942c1f7dc788a076d78220da4998a01bc9a631a78d9ca7bd6fc"} Oct 29 22:32:10.251174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.251079 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.251418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.251387 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:10.280161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.280122 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.280460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.280430 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:10.296120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.296082 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.296387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.296364 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:10.438765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.438734 5367 generic.go:356] "Generic (PLEG): container finished" podID="da9728a3-eefe-42db-bd84-e437b37ef405" containerID="50bfbfc8bcb0db32c61fce5f5d6ef5ad1da449dff7721c3042b2283d66889dab" exitCode=0 Oct 29 22:32:10.439159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:10.438946 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" event={"ID":"da9728a3-eefe-42db-bd84-e437b37ef405","Type":"ContainerDied","Data":"50bfbfc8bcb0db32c61fce5f5d6ef5ad1da449dff7721c3042b2283d66889dab"} Oct 29 22:32:11.251144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.251099 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:32:11.251310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.251180 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 22:32:11.277514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.277474 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:32:11.277678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.277539 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 22:32:11.293273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.293211 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:32:11.293401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.293274 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 29 22:32:11.643355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.643324 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:11.746758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.746386 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x\" (UniqueName: \"kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x\") pod \"da9728a3-eefe-42db-bd84-e437b37ef405\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " Oct 29 22:32:11.746758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.746537 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca\") pod \"da9728a3-eefe-42db-bd84-e437b37ef405\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " Oct 29 22:32:11.746758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.746626 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca\") pod \"da9728a3-eefe-42db-bd84-e437b37ef405\" (UID: \"da9728a3-eefe-42db-bd84-e437b37ef405\") " Oct 29 22:32:11.747929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.747847 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca" pod "da9728a3-eefe-42db-bd84-e437b37ef405" (UID: "da9728a3-eefe-42db-bd84-e437b37ef405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:11.749349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.749326 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:11.760122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.760039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca" pod "da9728a3-eefe-42db-bd84-e437b37ef405" (UID: "da9728a3-eefe-42db-bd84-e437b37ef405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:11.760520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.760480 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x" pod "da9728a3-eefe-42db-bd84-e437b37ef405" (UID: "da9728a3-eefe-42db-bd84-e437b37ef405"). InnerVolumeSpecName "kube-api-access-mjm9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:11.850720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.850668 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjm9x\" (UniqueName: \"kubernetes.io/projected/da9728a3-eefe-42db-bd84-e437b37ef405-kube-api-access-mjm9x\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:11.850895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:11.850735 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9728a3-eefe-42db-bd84-e437b37ef405-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:12.163188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.163142 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz"] Oct 29 22:32:12.164230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.164203 5367 cpu_manager.go:414] "containerMap: removing container" podUID="da9728a3-eefe-42db-bd84-e437b37ef405" containerName="revoke-keys" Oct 29 22:32:12.164230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.164230 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9728a3-eefe-42db-bd84-e437b37ef405" containerName="revoke-keys" Oct 29 22:32:12.164455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.164440 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9728a3-eefe-42db-bd84-e437b37ef405" containerName="revoke-keys" Oct 29 22:32:12.172158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.172136 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.250449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.250400 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz"] Oct 29 22:32:12.280766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.265388 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl65q\" (UniqueName: \"kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.280766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.265521 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.280766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.265577 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.387789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.387747 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl65q\" (UniqueName: \"kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.387973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.387870 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.387973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.387916 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.389894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.389869 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.392887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.392865 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.413377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.413318 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl65q\" (UniqueName: \"kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q\") pod \"maas-api-revoke-sub-1c66bd32-dttvz\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.525259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.521758 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" Oct 29 22:32:12.527479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.527443 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl" event={"ID":"da9728a3-eefe-42db-bd84-e437b37ef405","Type":"ContainerDied","Data":"ba2359e6517be942c1f7dc788a076d78220da4998a01bc9a631a78d9ca7bd6fc"} Oct 29 22:32:12.527656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.527641 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba2359e6517be942c1f7dc788a076d78220da4998a01bc9a631a78d9ca7bd6fc" Oct 29 22:32:12.622998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.622961 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:12.867718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.867010 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:12.869681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.868603 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:12.871332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.871257 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:32:12.891478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.891438 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:12.892776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.892740 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:12.894693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.894638 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:32:12.948234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.948205 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:12.948813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.948752 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:12.949734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.949690 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:32:12.974477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:12.974431 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz"] Oct 29 22:32:12.976134 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:12.976108 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e36897f_7dc0_484f_bf6c_81f1a960c428.slice/crio-0aeca430b8e6aad02a7783f1f80189d750505ac8430326dc1e0a1b244bc4ad71 WatchSource:0}: Error finding container 0aeca430b8e6aad02a7783f1f80189d750505ac8430326dc1e0a1b244bc4ad71: Status 404 returned error can't find the container with id 0aeca430b8e6aad02a7783f1f80189d750505ac8430326dc1e0a1b244bc4ad71 Oct 29 22:32:13.529993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:13.529954 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" event={"ID":"4e36897f-7dc0-484f-bf6c-81f1a960c428","Type":"ContainerStarted","Data":"0ad892a3a09c4cc3c893537d1adb29c08381b84af7249334f84c436c12ec85de"} Oct 29 22:32:13.530191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:13.530008 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" event={"ID":"4e36897f-7dc0-484f-bf6c-81f1a960c428","Type":"ContainerStarted","Data":"0aeca430b8e6aad02a7783f1f80189d750505ac8430326dc1e0a1b244bc4ad71"} Oct 29 22:32:13.574381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:13.574334 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" podStartSLOduration=1.574320384 podStartE2EDuration="1.574320384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:13.569692467 +0000 UTC m=+1000.463767388" watchObservedRunningTime="2026-10-29 22:32:13.574320384 +0000 UTC m=+1000.468395302" Oct 29 22:32:14.538829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.538791 5367 generic.go:356] "Generic (PLEG): container finished" podID="4e36897f-7dc0-484f-bf6c-81f1a960c428" containerID="0ad892a3a09c4cc3c893537d1adb29c08381b84af7249334f84c436c12ec85de" exitCode=0 Oct 29 22:32:14.539307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.538864 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" event={"ID":"4e36897f-7dc0-484f-bf6c-81f1a960c428","Type":"ContainerDied","Data":"0ad892a3a09c4cc3c893537d1adb29c08381b84af7249334f84c436c12ec85de"} Oct 29 22:32:14.865721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.847758 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd"] Oct 29 22:32:14.881503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.881469 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd"] Oct 29 22:32:14.881663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.881579 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:14.969982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.969950 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:14.970190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.970004 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcshw\" (UniqueName: \"kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:14.970190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:14.970099 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.071618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.071555 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.071861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.071646 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcshw\" (UniqueName: \"kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.071861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.071718 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.072994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.072936 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.074873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.074850 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.092785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.092752 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcshw\" (UniqueName: \"kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw\") pod \"maas-api-revoke-sub-fa563490-shscd\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.198709 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.198606 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:15.513879 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:15.513844 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0fb474_5b3b_450f_8f1e_697e64bf9b70.slice/crio-0ce3236401f9fd81d8a2788677c83f4c32f30c1c526541009d0573fe76a1bd6f WatchSource:0}: Error finding container 0ce3236401f9fd81d8a2788677c83f4c32f30c1c526541009d0573fe76a1bd6f: Status 404 returned error can't find the container with id 0ce3236401f9fd81d8a2788677c83f4c32f30c1c526541009d0573fe76a1bd6f Oct 29 22:32:15.514340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.514311 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd"] Oct 29 22:32:15.547725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.547654 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" event={"ID":"7f0fb474-5b3b-450f-8f1e-697e64bf9b70","Type":"ContainerStarted","Data":"0ce3236401f9fd81d8a2788677c83f4c32f30c1c526541009d0573fe76a1bd6f"} Oct 29 22:32:15.691389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.691363 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:15.786380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.786349 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca\") pod \"4e36897f-7dc0-484f-bf6c-81f1a960c428\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " Oct 29 22:32:15.786622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.786536 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca\") pod \"4e36897f-7dc0-484f-bf6c-81f1a960c428\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " Oct 29 22:32:15.786785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.786690 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q\" (UniqueName: \"kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q\") pod \"4e36897f-7dc0-484f-bf6c-81f1a960c428\" (UID: \"4e36897f-7dc0-484f-bf6c-81f1a960c428\") " Oct 29 22:32:15.787015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.786990 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca" pod "4e36897f-7dc0-484f-bf6c-81f1a960c428" (UID: "4e36897f-7dc0-484f-bf6c-81f1a960c428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:15.787340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.787318 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:15.788604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.788580 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca" pod "4e36897f-7dc0-484f-bf6c-81f1a960c428" (UID: "4e36897f-7dc0-484f-bf6c-81f1a960c428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:15.788690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.788675 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q" pod "4e36897f-7dc0-484f-bf6c-81f1a960c428" (UID: "4e36897f-7dc0-484f-bf6c-81f1a960c428"). InnerVolumeSpecName "kube-api-access-pl65q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:15.888217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.888180 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e36897f-7dc0-484f-bf6c-81f1a960c428-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:15.888217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:15.888210 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl65q\" (UniqueName: \"kubernetes.io/projected/4e36897f-7dc0-484f-bf6c-81f1a960c428-kube-api-access-pl65q\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:16.557725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:16.557659 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" Oct 29 22:32:16.558185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:16.557716 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz" event={"ID":"4e36897f-7dc0-484f-bf6c-81f1a960c428","Type":"ContainerDied","Data":"0aeca430b8e6aad02a7783f1f80189d750505ac8430326dc1e0a1b244bc4ad71"} Oct 29 22:32:16.558185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:16.557771 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aeca430b8e6aad02a7783f1f80189d750505ac8430326dc1e0a1b244bc4ad71" Oct 29 22:32:16.559540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:16.559517 5367 generic.go:356] "Generic (PLEG): container finished" podID="7f0fb474-5b3b-450f-8f1e-697e64bf9b70" containerID="7303089e2f05b77b183812191ebf5dd4c437dbcd0e30166bf8a120258913ee1e" exitCode=0 Oct 29 22:32:16.559647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:16.559563 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" event={"ID":"7f0fb474-5b3b-450f-8f1e-697e64bf9b70","Type":"ContainerDied","Data":"7303089e2f05b77b183812191ebf5dd4c437dbcd0e30166bf8a120258913ee1e"} Oct 29 22:32:17.712740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.712691 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:17.814507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.814454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw\" (UniqueName: \"kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw\") pod \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " Oct 29 22:32:17.814678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.814644 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca\") pod \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " Oct 29 22:32:17.814822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.814809 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca\") pod \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\" (UID: \"7f0fb474-5b3b-450f-8f1e-697e64bf9b70\") " Oct 29 22:32:17.814980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.814956 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca" pod "7f0fb474-5b3b-450f-8f1e-697e64bf9b70" (UID: "7f0fb474-5b3b-450f-8f1e-697e64bf9b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:17.815295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.815277 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:17.816603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.816571 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw" pod "7f0fb474-5b3b-450f-8f1e-697e64bf9b70" (UID: "7f0fb474-5b3b-450f-8f1e-697e64bf9b70"). InnerVolumeSpecName "kube-api-access-bcshw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:17.816969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.816948 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca" pod "7f0fb474-5b3b-450f-8f1e-697e64bf9b70" (UID: "7f0fb474-5b3b-450f-8f1e-697e64bf9b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:17.916307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.916272 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:17.916307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:17.916301 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcshw\" (UniqueName: \"kubernetes.io/projected/7f0fb474-5b3b-450f-8f1e-697e64bf9b70-kube-api-access-bcshw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:18.428858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.428823 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j"] Oct 29 22:32:18.429430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.429417 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4e36897f-7dc0-484f-bf6c-81f1a960c428" containerName="revoke-keys" Oct 29 22:32:18.429474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.429431 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e36897f-7dc0-484f-bf6c-81f1a960c428" containerName="revoke-keys" Oct 29 22:32:18.429474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.429452 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7f0fb474-5b3b-450f-8f1e-697e64bf9b70" containerName="revoke-keys" Oct 29 22:32:18.429474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.429457 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0fb474-5b3b-450f-8f1e-697e64bf9b70" containerName="revoke-keys" Oct 29 22:32:18.429565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.429558 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e36897f-7dc0-484f-bf6c-81f1a960c428" containerName="revoke-keys" Oct 29 22:32:18.429596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.429568 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0fb474-5b3b-450f-8f1e-697e64bf9b70" containerName="revoke-keys" Oct 29 22:32:18.433006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.432990 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.475507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.475468 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j"] Oct 29 22:32:18.524802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.524766 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.524802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.524809 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.524916 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.525092 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.525209 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jzq\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.525291 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.525344 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.525400 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.525485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.525465 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.577562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.577533 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" Oct 29 22:32:18.577765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.577563 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd" event={"ID":"7f0fb474-5b3b-450f-8f1e-697e64bf9b70","Type":"ContainerDied","Data":"0ce3236401f9fd81d8a2788677c83f4c32f30c1c526541009d0573fe76a1bd6f"} Oct 29 22:32:18.577765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.577615 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce3236401f9fd81d8a2788677c83f4c32f30c1c526541009d0573fe76a1bd6f" Oct 29 22:32:18.626574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626549 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.626716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626580 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.626716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626612 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.626716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626680 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.626894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626753 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jzq\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.626894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626800 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.626894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626830 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626895 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626977 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.626992 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.627047 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.627275 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.627332 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.627482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.627457 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.629132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.629107 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.629537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.629513 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.647482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.647456 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.647637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.647620 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jzq\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.698111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.698042 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt"] Oct 29 22:32:18.704109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.704084 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.726196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.726170 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt"] Oct 29 22:32:18.730665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.730633 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.730825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.730740 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7sb\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.730894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.730835 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.730975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.730953 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.731149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.731131 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.731283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.731184 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.731380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.731339 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.731466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.731377 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.731466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.731419 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.748045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.747567 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:18.833372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833329 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833389 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833478 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833508 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833534 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833589 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833652 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7sb\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833684 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.833921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.833839 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.834428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.834380 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.834590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.834394 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.836311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.835226 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.836311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.835989 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.836311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.836011 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.836719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.836673 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.841022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.840973 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.855181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.855134 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7sb\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:18.855827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:18.855803 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:19.043453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.043369 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:19.133843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.133811 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl"] Oct 29 22:32:19.144214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.144069 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j"] Oct 29 22:32:19.144214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.144184 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.147043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.146999 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.147143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.147088 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.147143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.147135 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.159738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.159712 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl"] Oct 29 22:32:19.239426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239284 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnk6\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.239426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239335 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.239581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239462 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.239581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239521 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.239785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239726 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.239862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239841 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.239923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.239910 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.240082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.240043 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.240224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.240103 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.341952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.341919 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.341977 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342040 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342105 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342165 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnk6\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342202 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342240 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342284 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342431 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342507 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342538 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342820 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.342983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.342960 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.343125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.343103 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.344681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.344659 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.345128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.345111 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.354262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.354237 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt"] Oct 29 22:32:19.354385 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:19.354364 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8497a33f_5655_4a84_aa30_b98c024e9645.slice/crio-bba1a9f71f535f90d48ddc6477c72cdd37a8125d6c1aa0e1d8509e4b5dcd7be2 WatchSource:0}: Error finding container bba1a9f71f535f90d48ddc6477c72cdd37a8125d6c1aa0e1d8509e4b5dcd7be2: Status 404 returned error can't find the container with id bba1a9f71f535f90d48ddc6477c72cdd37a8125d6c1aa0e1d8509e4b5dcd7be2 Oct 29 22:32:19.357008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.356976 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.357107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.357053 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.357107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.357098 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.362167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.362136 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.362535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.362515 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnk6\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.466574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.466531 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:19.597502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.597406 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" event={"ID":"8497a33f-5655-4a84-aa30-b98c024e9645","Type":"ContainerStarted","Data":"32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10"} Oct 29 22:32:19.597502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.597494 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" event={"ID":"8497a33f-5655-4a84-aa30-b98c024e9645","Type":"ContainerStarted","Data":"bba1a9f71f535f90d48ddc6477c72cdd37a8125d6c1aa0e1d8509e4b5dcd7be2"} Oct 29 22:32:19.599930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.599895 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" event={"ID":"1a6181a8-4db4-46bf-8c30-559987db96ae","Type":"ContainerStarted","Data":"bd06dcdc99e9023ee288ca2750b9dc6bc36d216f881ccc520c3f754322c9bc09"} Oct 29 22:32:19.600027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.599965 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" event={"ID":"1a6181a8-4db4-46bf-8c30-559987db96ae","Type":"ContainerStarted","Data":"5923953edbf21aee02ca754f1afb1919b95850fd143e367425797c35c58fb447"} Oct 29 22:32:19.644919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.644869 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podStartSLOduration=1.6448558580000001 podStartE2EDuration="1.644855858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:19.640434868 +0000 UTC m=+1006.534509788" watchObservedRunningTime="2026-10-29 22:32:19.644855858 +0000 UTC m=+1006.538930776" Oct 29 22:32:19.687103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.687039 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podStartSLOduration=1.687025532 podStartE2EDuration="1.687025532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:19.68609479 +0000 UTC m=+1006.580169710" watchObservedRunningTime="2026-10-29 22:32:19.687025532 +0000 UTC m=+1006.581100455" Oct 29 22:32:19.747813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.747778 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:19.788376 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:19.788334 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be98a21_c1ba_4ecf_9678_330ec53a8106.slice/crio-09ee2a2f2e4add1739e6fdeaba283ca0fc9d37ed7c6b442fec7fc8625e3bfb8c WatchSource:0}: Error finding container 09ee2a2f2e4add1739e6fdeaba283ca0fc9d37ed7c6b442fec7fc8625e3bfb8c: Status 404 returned error can't find the container with id 09ee2a2f2e4add1739e6fdeaba283ca0fc9d37ed7c6b442fec7fc8625e3bfb8c Oct 29 22:32:19.789068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.789044 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl"] Oct 29 22:32:19.791058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.791026 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.791126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.791088 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:19.791126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:19.791117 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:20.044393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.044365 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:20.046134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.046106 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.046237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.046158 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:20.194892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.194809 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd"] Oct 29 22:32:20.220120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.220091 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.227830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.227763 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd"] Oct 29 22:32:20.238148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.238123 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:32:20.238279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.238198 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:20.238279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.238241 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:32:20.238279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.238256 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:20.316157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.316118 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.316356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.316210 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbhr\" (UniqueName: \"kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.316356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.316272 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.418427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.418385 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.418626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.418450 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbhr\" (UniqueName: \"kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.418626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.418509 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.419203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.419182 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.426746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.421433 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.439750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.439721 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbhr\" (UniqueName: \"kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr\") pod \"maas-api-revoke-sub-63aedb82-5c6zd\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.543321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.543017 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:20.640903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.640857 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" event={"ID":"9be98a21-c1ba-4ecf-9678-330ec53a8106","Type":"ContainerStarted","Data":"a8ef8bd1ce4900a1907a1b438b06625cc686f4064c84d4c1e81ccfdaabc3c8c9"} Oct 29 22:32:20.641095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.640924 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" event={"ID":"9be98a21-c1ba-4ecf-9678-330ec53a8106","Type":"ContainerStarted","Data":"09ee2a2f2e4add1739e6fdeaba283ca0fc9d37ed7c6b442fec7fc8625e3bfb8c"} Oct 29 22:32:20.736957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.736887 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podStartSLOduration=1.736864159 podStartE2EDuration="1.736864159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:20.71866946 +0000 UTC m=+1007.612744387" watchObservedRunningTime="2026-10-29 22:32:20.736864159 +0000 UTC m=+1007.630939087" Oct 29 22:32:20.749270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.749238 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.749652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:20.749296 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 29 22:32:21.067722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.054744 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.067722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.054804 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:21.350085 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:21.350043 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad0113a_2752_4a4a_92a3_8eca635a4f9f.slice/crio-4ddee096f1006e87ca3a5fda2e05ab423361969ded45291ba2ebbd185ca8d48f WatchSource:0}: Error finding container 4ddee096f1006e87ca3a5fda2e05ab423361969ded45291ba2ebbd185ca8d48f: Status 404 returned error can't find the container with id 4ddee096f1006e87ca3a5fda2e05ab423361969ded45291ba2ebbd185ca8d48f Oct 29 22:32:21.471132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.452798 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd"] Oct 29 22:32:21.494724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.487182 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:21.517726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.516860 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.517726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.516935 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:32:21.651201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.651163 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" event={"ID":"8ad0113a-2752-4a4a-92a3-8eca635a4f9f","Type":"ContainerStarted","Data":"4ddee096f1006e87ca3a5fda2e05ab423361969ded45291ba2ebbd185ca8d48f"} Oct 29 22:32:21.748954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.748921 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.749125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:21.748982 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 29 22:32:22.045201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.045165 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.045780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.045746 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:22.437090 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:32:22.436466 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:32:22.475717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.468032 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.475717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.468090 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:32:22.739617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.739533 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" event={"ID":"8ad0113a-2752-4a4a-92a3-8eca635a4f9f","Type":"ContainerStarted","Data":"df5fa6a383eb4cc1eaefdf6e4bb4a7b93ca9c52fe7f4a601532758991eeb2f7b"} Oct 29 22:32:22.748520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.748483 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.748772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:22.748534 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 29 22:32:23.044308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.044198 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.044308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.044269 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:23.453179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.453117 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" podStartSLOduration=3.453095922 podStartE2EDuration="3.453095922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:22.807132951 +0000 UTC m=+1009.701207903" watchObservedRunningTime="2026-10-29 22:32:23.453095922 +0000 UTC m=+1010.347170843" Oct 29 22:32:23.503762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.467070 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.503762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.467129 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:32:23.548215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.548164 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl"] Oct 29 22:32:23.553195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.553173 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl"] Oct 29 22:32:23.553350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.553250 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.574721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.573048 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"distinct-2-w0-40cf9a-kserve-self-signed-certs\"" Oct 29 22:32:23.706773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.684447 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.722724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.714029 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.739829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.739794 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.740040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.740024 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.740173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.740155 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.740308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.740292 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6p2g\" (UniqueName: \"kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.748484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.748456 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.748734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.748679 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 29 22:32:23.757746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.753723 5367 generic.go:356] "Generic (PLEG): container finished" podID="8ad0113a-2752-4a4a-92a3-8eca635a4f9f" containerID="df5fa6a383eb4cc1eaefdf6e4bb4a7b93ca9c52fe7f4a601532758991eeb2f7b" exitCode=0 Oct 29 22:32:23.761724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.757942 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" event={"ID":"8ad0113a-2752-4a4a-92a3-8eca635a4f9f","Type":"ContainerDied","Data":"df5fa6a383eb4cc1eaefdf6e4bb4a7b93ca9c52fe7f4a601532758991eeb2f7b"} Oct 29 22:32:23.871499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.871460 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.871859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.871831 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.894862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.894819 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6p2g\" (UniqueName: \"kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.911729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.895243 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.935785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.912245 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.953754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.944811 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.970808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.872574 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:23.999721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.995802 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:24.074945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:23.918812 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:24.074945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.045039 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.074945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.072865 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:24.183462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.183428 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6p2g\" (UniqueName: \"kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:24.218892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.218855 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:24.232877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.232795 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs\") pod \"distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:24.314728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.314676 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:24.528907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.528817 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.528907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.528880 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:32:24.749219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.749085 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.749219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:24.749146 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 29 22:32:25.144035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.143997 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.144312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.144278 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:25.350581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.350496 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv"] Oct 29 22:32:25.361298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.361271 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.416723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.395579 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:32:25.430995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.419063 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cvlmz\"" Oct 29 22:32:25.430995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.419096 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:32:25.431161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.431045 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:32:25.437161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.436771 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv"] Oct 29 22:32:25.441869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.441631 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl"] Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453129 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453176 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453214 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453264 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453307 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453363 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgfl\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.457735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.453396 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.486048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.486015 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.486300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.486275 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:32:25.579754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.579691 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.589056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.589031 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.589290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.589258 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.592844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.592821 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.593119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.593095 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.593326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.593310 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.593588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.593543 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgfl\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.595601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.595578 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.601232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.601143 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.601232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.601201 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.602669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.602648 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.602836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.602722 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.614568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.614542 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.615597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.615554 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgfl\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xc4fv\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.687189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.686768 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:25.695364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.695332 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca\") pod \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " Oct 29 22:32:25.695511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.695385 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr\" (UniqueName: \"kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr\") pod \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " Oct 29 22:32:25.695511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.695429 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca\") pod \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\" (UID: \"8ad0113a-2752-4a4a-92a3-8eca635a4f9f\") " Oct 29 22:32:25.695999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.695687 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca" pod "8ad0113a-2752-4a4a-92a3-8eca635a4f9f" (UID: "8ad0113a-2752-4a4a-92a3-8eca635a4f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:25.696117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.696050 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:25.699299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.699272 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca" pod "8ad0113a-2752-4a4a-92a3-8eca635a4f9f" (UID: "8ad0113a-2752-4a4a-92a3-8eca635a4f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:25.700134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.700113 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr" pod "8ad0113a-2752-4a4a-92a3-8eca635a4f9f" (UID: "8ad0113a-2752-4a4a-92a3-8eca635a4f9f"). InnerVolumeSpecName "kube-api-access-cjbhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:25.739852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.739817 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:25.748560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.748532 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.748677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.748588 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 29 22:32:25.798157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.798127 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbhr\" (UniqueName: \"kubernetes.io/projected/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-kube-api-access-cjbhr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:25.798278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:25.798164 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad0113a-2752-4a4a-92a3-8eca635a4f9f-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:26.045214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.045179 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.045389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.045250 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 29 22:32:26.108759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.106058 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" event={"ID":"0eb12b49-8fc5-4d29-8dd3-8016cc31162f","Type":"ContainerStarted","Data":"0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd"} Oct 29 22:32:26.108759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.106107 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" event={"ID":"0eb12b49-8fc5-4d29-8dd3-8016cc31162f","Type":"ContainerStarted","Data":"fed73fa13506aa3ff56afdb742a2a7810d9d92e94c876c1dfd4d4eda8f2d0534"} Oct 29 22:32:26.108759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.107721 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" event={"ID":"8ad0113a-2752-4a4a-92a3-8eca635a4f9f","Type":"ContainerDied","Data":"4ddee096f1006e87ca3a5fda2e05ab423361969ded45291ba2ebbd185ca8d48f"} Oct 29 22:32:26.108759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.107769 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ddee096f1006e87ca3a5fda2e05ab423361969ded45291ba2ebbd185ca8d48f" Oct 29 22:32:26.108759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.107822 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd" Oct 29 22:32:26.121810 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:26.121771 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bb90e7_c597_4a56_a31c_264a2387b1ca.slice/crio-2ab285f599bf4ad79453df5055775010f33fda5265fe2689a6c6399ca4b67c60 WatchSource:0}: Error finding container 2ab285f599bf4ad79453df5055775010f33fda5265fe2689a6c6399ca4b67c60: Status 404 returned error can't find the container with id 2ab285f599bf4ad79453df5055775010f33fda5265fe2689a6c6399ca4b67c60 Oct 29 22:32:26.127941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.127915 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv"] Oct 29 22:32:26.188754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.185335 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podStartSLOduration=3.185315115 podStartE2EDuration="3.185315115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:26.172060526 +0000 UTC m=+1013.066135447" watchObservedRunningTime="2026-10-29 22:32:26.185315115 +0000 UTC m=+1013.079390036" Oct 29 22:32:26.324465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.324434 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:26.467302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.467218 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.467302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:26.467284 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:32:27.088008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.087978 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c"] Oct 29 22:32:27.089935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.089914 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8ad0113a-2752-4a4a-92a3-8eca635a4f9f" containerName="revoke-keys" Oct 29 22:32:27.090092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.090080 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad0113a-2752-4a4a-92a3-8eca635a4f9f" containerName="revoke-keys" Oct 29 22:32:27.094080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.094063 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad0113a-2752-4a4a-92a3-8eca635a4f9f" containerName="revoke-keys" Oct 29 22:32:27.125092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.114456 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c"] Oct 29 22:32:27.125092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.114596 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.143681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.143653 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wmq\" (UniqueName: \"kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.159712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.144056 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:27.159712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.144104 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:27.159712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.144176 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:32:27.159712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.144237 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:32:27.162971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.162947 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.163445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.163427 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.164029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.164007 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" event={"ID":"49bb90e7-c597-4a56-a31c-264a2387b1ca","Type":"ContainerStarted","Data":"2ab285f599bf4ad79453df5055775010f33fda5265fe2689a6c6399ca4b67c60"} Oct 29 22:32:27.267576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.266344 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.267576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.266441 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.267576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.266717 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wmq\" (UniqueName: \"kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.269783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.269024 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.293495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.293447 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wmq\" (UniqueName: \"kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.326164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.326109 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d21112-6ds5c\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.463455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.463154 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:27.490990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.489004 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:27.498728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.497630 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:27.499913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.499681 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:32:27.585363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.584107 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:27.590043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.589386 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:27.596765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.593185 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:32:27.785265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.785160 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:27.785974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.785953 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:27.787730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.787690 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:32:27.882920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:27.882892 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c"] Oct 29 22:32:27.883417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:27.883380 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13924bfc_cde2_4e61_ba68_539f640ad853.slice/crio-938b6a77b397bd820468247b90b6eadf18c73df4a74b0342b5b357e017b4709f WatchSource:0}: Error finding container 938b6a77b397bd820468247b90b6eadf18c73df4a74b0342b5b357e017b4709f: Status 404 returned error can't find the container with id 938b6a77b397bd820468247b90b6eadf18c73df4a74b0342b5b357e017b4709f Oct 29 22:32:28.176415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:28.176381 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" event={"ID":"13924bfc-cde2-4e61-ba68-539f640ad853","Type":"ContainerStarted","Data":"938b6a77b397bd820468247b90b6eadf18c73df4a74b0342b5b357e017b4709f"} Oct 29 22:32:29.118873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.118838 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9"] Oct 29 22:32:29.124280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.124253 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.152869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.152790 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9"] Oct 29 22:32:29.191920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.191878 5367 generic.go:356] "Generic (PLEG): container finished" podID="13924bfc-cde2-4e61-ba68-539f640ad853" containerID="292f2736ae873edf9082be48935e844cefc86f1f7620dfd0a7999a1f49ebe7b0" exitCode=0 Oct 29 22:32:29.193379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.193354 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" event={"ID":"13924bfc-cde2-4e61-ba68-539f640ad853","Type":"ContainerDied","Data":"292f2736ae873edf9082be48935e844cefc86f1f7620dfd0a7999a1f49ebe7b0"} Oct 29 22:32:29.201112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.201081 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47w89\" (UniqueName: \"kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.201292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.201250 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.201490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.201469 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.302976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.302926 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.303148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.303053 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47w89\" (UniqueName: \"kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.303208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.303189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.304099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.304072 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.306764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.306730 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.341260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.341237 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47w89\" (UniqueName: \"kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89\") pod \"maas-api-revoke-sub-e1618b90-sjnf9\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.441114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.441029 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:29.710624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.710589 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:29.710746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.710657 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:29.769602 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:29.769569 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d84bbb7_c2f8_49b9_9116_5adb01f9db12.slice/crio-0d775bd293b0ec3ec2d2dc75fb21507322f6ffffee3ece7dec1be5444e0c01a4 WatchSource:0}: Error finding container 0d775bd293b0ec3ec2d2dc75fb21507322f6ffffee3ece7dec1be5444e0c01a4: Status 404 returned error can't find the container with id 0d775bd293b0ec3ec2d2dc75fb21507322f6ffffee3ece7dec1be5444e0c01a4 Oct 29 22:32:29.770528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:29.770505 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9"] Oct 29 22:32:30.218607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.218558 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" event={"ID":"7d84bbb7-c2f8-49b9-9116-5adb01f9db12","Type":"ContainerStarted","Data":"546524d927559c3e4e606bb59352ce166e839900dc1de497c3b282ee1dfd64de"} Oct 29 22:32:30.219162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.218628 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" event={"ID":"7d84bbb7-c2f8-49b9-9116-5adb01f9db12","Type":"ContainerStarted","Data":"0d775bd293b0ec3ec2d2dc75fb21507322f6ffffee3ece7dec1be5444e0c01a4"} Oct 29 22:32:30.228834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.228634 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" event={"ID":"49bb90e7-c597-4a56-a31c-264a2387b1ca","Type":"ContainerStarted","Data":"13bc1d41b2a1d4cd5e22f4bd3bce9814fb2fbd4863c5e7f4b28b8a1939a02c16"} Oct 29 22:32:30.228834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.228689 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:30.242753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.231314 5367 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xc4fv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:32:30.242753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.231373 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" podUID="49bb90e7-c597-4a56-a31c-264a2387b1ca" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:32:30.266570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.266516 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" podStartSLOduration=1.266497189 podStartE2EDuration="1.266497189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:30.262513055 +0000 UTC m=+1017.156587977" watchObservedRunningTime="2026-10-29 22:32:30.266497189 +0000 UTC m=+1017.160572111" Oct 29 22:32:30.451383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.451345 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:30.492281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.492179 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" podStartSLOduration=1.9075086049999999 podStartE2EDuration="5.492161104s" totalImagesPullingTime="3.584652499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:30.324292385 +0000 UTC m=+1017.218367305" watchObservedRunningTime="2026-10-29 22:32:30.492161104 +0000 UTC m=+1017.386236024" Oct 29 22:32:30.532746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.532665 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca\") pod \"13924bfc-cde2-4e61-ba68-539f640ad853\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " Oct 29 22:32:30.533196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.533169 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq\" (UniqueName: \"kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq\") pod \"13924bfc-cde2-4e61-ba68-539f640ad853\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " Oct 29 22:32:30.533321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.533223 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca" pod "13924bfc-cde2-4e61-ba68-539f640ad853" (UID: "13924bfc-cde2-4e61-ba68-539f640ad853"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.533878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.533852 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca\") pod \"13924bfc-cde2-4e61-ba68-539f640ad853\" (UID: \"13924bfc-cde2-4e61-ba68-539f640ad853\") " Oct 29 22:32:30.534430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.534396 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.535876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.535849 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq" pod "13924bfc-cde2-4e61-ba68-539f640ad853" (UID: "13924bfc-cde2-4e61-ba68-539f640ad853"). InnerVolumeSpecName "kube-api-access-p9wmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:30.540640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.540609 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca" pod "13924bfc-cde2-4e61-ba68-539f640ad853" (UID: "13924bfc-cde2-4e61-ba68-539f640ad853"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:30.635881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.635829 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wmq\" (UniqueName: \"kubernetes.io/projected/13924bfc-cde2-4e61-ba68-539f640ad853-kube-api-access-p9wmq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.635881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:30.635876 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13924bfc-cde2-4e61-ba68-539f640ad853-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:31.237487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:31.237461 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" Oct 29 22:32:31.237926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:31.237495 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c" event={"ID":"13924bfc-cde2-4e61-ba68-539f640ad853","Type":"ContainerDied","Data":"938b6a77b397bd820468247b90b6eadf18c73df4a74b0342b5b357e017b4709f"} Oct 29 22:32:31.237926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:31.237552 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="938b6a77b397bd820468247b90b6eadf18c73df4a74b0342b5b357e017b4709f" Oct 29 22:32:31.239757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:31.239732 5367 generic.go:356] "Generic (PLEG): container finished" podID="7d84bbb7-c2f8-49b9-9116-5adb01f9db12" containerID="546524d927559c3e4e606bb59352ce166e839900dc1de497c3b282ee1dfd64de" exitCode=0 Oct 29 22:32:31.240225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:31.240188 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" event={"ID":"7d84bbb7-c2f8-49b9-9116-5adb01f9db12","Type":"ContainerDied","Data":"546524d927559c3e4e606bb59352ce166e839900dc1de497c3b282ee1dfd64de"} Oct 29 22:32:31.241947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:31.241928 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 22:32:32.444264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.444237 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:32.577335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.577302 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca\") pod \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " Oct 29 22:32:32.577511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.577475 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca\") pod \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " Oct 29 22:32:32.577588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.577570 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89\" (UniqueName: \"kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89\") pod \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\" (UID: \"7d84bbb7-c2f8-49b9-9116-5adb01f9db12\") " Oct 29 22:32:32.577819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.577796 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca" pod "7d84bbb7-c2f8-49b9-9116-5adb01f9db12" (UID: "7d84bbb7-c2f8-49b9-9116-5adb01f9db12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:32.578295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.578271 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:32.579607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.579577 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca" pod "7d84bbb7-c2f8-49b9-9116-5adb01f9db12" (UID: "7d84bbb7-c2f8-49b9-9116-5adb01f9db12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:32.579812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.579693 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89" pod "7d84bbb7-c2f8-49b9-9116-5adb01f9db12" (UID: "7d84bbb7-c2f8-49b9-9116-5adb01f9db12"). InnerVolumeSpecName "kube-api-access-47w89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:32.680304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.680257 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:32.680304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:32.680301 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47w89\" (UniqueName: \"kubernetes.io/projected/7d84bbb7-c2f8-49b9-9116-5adb01f9db12-kube-api-access-47w89\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:33.402790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:33.398915 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" event={"ID":"7d84bbb7-c2f8-49b9-9116-5adb01f9db12","Type":"ContainerDied","Data":"0d775bd293b0ec3ec2d2dc75fb21507322f6ffffee3ece7dec1be5444e0c01a4"} Oct 29 22:32:33.402790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:33.398988 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d775bd293b0ec3ec2d2dc75fb21507322f6ffffee3ece7dec1be5444e0c01a4" Oct 29 22:32:33.402790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:33.399071 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9" Oct 29 22:32:34.340722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:34.339236 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:34.716800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:34.715786 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.815732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:34.809785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.911731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:34.906785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.017776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.000714 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.017776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.001022 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.017776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.010631 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.035732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.034049 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:32:35.129722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.128779 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.288798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.281845 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.288798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.283873 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.348733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.340779 5367 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.348733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.341091 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.578721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.577517 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:32:35.696759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.684910 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.737223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.737083 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.808738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:35.773331 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:32:36.094871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.061978 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:32:36.144728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.119982 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:32:36.240382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.238917 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:32:36.295429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.295397 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:32:36.380927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.347290 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.380927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.347726 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.380927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.362176 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.380927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.363418 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:32:36.380927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.363495 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:32:36.440721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.420036 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:32:36.492721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.459833 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:32:36.553727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.496769 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:32:36.553727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.498531 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:32:36.553727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.499320 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:32:36.553727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.499731 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:32:36.579252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.579209 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.706030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.705948 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:32:36.706030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.706015 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:32:36.706237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.706058 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:32:36.706237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.706088 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:32:36.707199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.707172 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:32:36.711854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.711833 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:32:36.718247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.718222 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:32:36.727737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.723626 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:32:36.727737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.724415 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:32:36.727737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.725776 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:32:36.732159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.732137 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:32:36.738405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.732544 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:32:36.753091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:36.753009 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:32:43.004116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004082 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j"] Oct 29 22:32:43.004783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004765 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7d84bbb7-c2f8-49b9-9116-5adb01f9db12" containerName="revoke-keys" Oct 29 22:32:43.004854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004784 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d84bbb7-c2f8-49b9-9116-5adb01f9db12" containerName="revoke-keys" Oct 29 22:32:43.004854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004821 5367 cpu_manager.go:414] "containerMap: removing container" podUID="13924bfc-cde2-4e61-ba68-539f640ad853" containerName="revoke-keys" Oct 29 22:32:43.004854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004826 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13924bfc-cde2-4e61-ba68-539f640ad853" containerName="revoke-keys" Oct 29 22:32:43.004952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004942 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d84bbb7-c2f8-49b9-9116-5adb01f9db12" containerName="revoke-keys" Oct 29 22:32:43.004984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.004960 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="13924bfc-cde2-4e61-ba68-539f640ad853" containerName="revoke-keys" Oct 29 22:32:43.010105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.010089 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.026433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.026403 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:43.026613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.026493 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:32:43.026613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.026548 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:32:43.026613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.026566 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:43.032970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.032790 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j"] Oct 29 22:32:43.088215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.088174 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.088215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.088230 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.088439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.088374 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n85g\" (UniqueName: \"kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.189414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.189380 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.189600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.189426 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.189600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.189505 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n85g\" (UniqueName: \"kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.190118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.190092 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.191746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.191729 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.226050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.226028 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n85g\" (UniqueName: \"kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g\") pod \"maas-api-revoke-sub-d85fccdc-khq8j\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.324862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.324836 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:43.640016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:43.629766 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j"] Oct 29 22:32:43.640016 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:43.635008 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e7a230_31aa_4b8d_80e5_78f9d0b1627d.slice/crio-cd3e7ba60d2dfad45153c057d2c9e337ba2c6278c2e2614e2616316fd73d61d5 WatchSource:0}: Error finding container cd3e7ba60d2dfad45153c057d2c9e337ba2c6278c2e2614e2616316fd73d61d5: Status 404 returned error can't find the container with id cd3e7ba60d2dfad45153c057d2c9e337ba2c6278c2e2614e2616316fd73d61d5 Oct 29 22:32:44.027106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:44.027074 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" event={"ID":"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d","Type":"ContainerStarted","Data":"afa4ef1d9dcdca984c1f5729f70f4ef9a12761871bd99c67d694d4cb5502d2c3"} Oct 29 22:32:44.027450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:44.027125 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" event={"ID":"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d","Type":"ContainerStarted","Data":"cd3e7ba60d2dfad45153c057d2c9e337ba2c6278c2e2614e2616316fd73d61d5"} Oct 29 22:32:44.070613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:44.070562 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" podStartSLOduration=2.0705495369999998 podStartE2EDuration="2.070549537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:44.067878312 +0000 UTC m=+1030.961953232" watchObservedRunningTime="2026-10-29 22:32:44.070549537 +0000 UTC m=+1030.964624458" Oct 29 22:32:44.336474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:44.336450 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:44.347554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:44.347532 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:32:45.037582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:45.037545 5367 generic.go:356] "Generic (PLEG): container finished" podID="d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" containerID="afa4ef1d9dcdca984c1f5729f70f4ef9a12761871bd99c67d694d4cb5502d2c3" exitCode=0 Oct 29 22:32:45.038016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:45.037616 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" event={"ID":"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d","Type":"ContainerDied","Data":"afa4ef1d9dcdca984c1f5729f70f4ef9a12761871bd99c67d694d4cb5502d2c3"} Oct 29 22:32:46.227577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.227542 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:46.341278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.341239 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca\") pod \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " Oct 29 22:32:46.341494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.341468 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca\") pod \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " Oct 29 22:32:46.341577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.341551 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g\" (UniqueName: \"kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g\") pod \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\" (UID: \"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d\") " Oct 29 22:32:46.342182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.342148 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca" pod "d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" (UID: "d7e7a230-31aa-4b8d-80e5-78f9d0b1627d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:46.364182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.364150 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:46.369228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.369160 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g" pod "d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" (UID: "d7e7a230-31aa-4b8d-80e5-78f9d0b1627d"). InnerVolumeSpecName "kube-api-access-9n85g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:46.381584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.375364 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca" pod "d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" (UID: "d7e7a230-31aa-4b8d-80e5-78f9d0b1627d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:46.465261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.465225 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:46.465428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.465281 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n85g\" (UniqueName: \"kubernetes.io/projected/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d-kube-api-access-9n85g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:46.895204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.895171 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6"] Oct 29 22:32:46.895866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.895847 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" containerName="revoke-keys" Oct 29 22:32:46.895866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.895863 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" containerName="revoke-keys" Oct 29 22:32:46.896041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.895990 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" containerName="revoke-keys" Oct 29 22:32:46.899821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.899801 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:46.918938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.918916 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6"] Oct 29 22:32:46.971791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.971760 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:46.971913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.971810 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplsf\" (UniqueName: \"kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:46.971913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:46.971891 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.054583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.054556 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" Oct 29 22:32:47.054736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.054593 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j" event={"ID":"d7e7a230-31aa-4b8d-80e5-78f9d0b1627d","Type":"ContainerDied","Data":"cd3e7ba60d2dfad45153c057d2c9e337ba2c6278c2e2614e2616316fd73d61d5"} Oct 29 22:32:47.054736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.054641 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd3e7ba60d2dfad45153c057d2c9e337ba2c6278c2e2614e2616316fd73d61d5" Oct 29 22:32:47.073252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.073221 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.073358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.073286 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplsf\" (UniqueName: \"kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.073358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.073339 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.073879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.073861 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.075724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.075691 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.096061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.096037 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplsf\" (UniqueName: \"kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf\") pod \"maas-api-revoke-sub-191ddbbe-pfjt6\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.218945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.218866 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:47.522158 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:47.522122 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ceac8e_6bd2_4a5d_92b6_7a2fb900df27.slice/crio-7195022d8588040ab577aa848e5ab53df8216446a9b18220af23093ba44297e0 WatchSource:0}: Error finding container 7195022d8588040ab577aa848e5ab53df8216446a9b18220af23093ba44297e0: Status 404 returned error can't find the container with id 7195022d8588040ab577aa848e5ab53df8216446a9b18220af23093ba44297e0 Oct 29 22:32:47.522974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:47.522946 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6"] Oct 29 22:32:48.076147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.076107 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" event={"ID":"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27","Type":"ContainerStarted","Data":"ae1aa5b6773aa3a07d0f4f1da4e375657a8df3c5d5ae10c7e8bc9fa723808a51"} Oct 29 22:32:48.076337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.076172 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" event={"ID":"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27","Type":"ContainerStarted","Data":"7195022d8588040ab577aa848e5ab53df8216446a9b18220af23093ba44297e0"} Oct 29 22:32:48.131533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.123513 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" podStartSLOduration=2.123494559 podStartE2EDuration="2.123494559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:48.115476157 +0000 UTC m=+1035.009551080" watchObservedRunningTime="2026-10-29 22:32:48.123494559 +0000 UTC m=+1035.017569482" Oct 29 22:32:48.494060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.494027 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7"] Oct 29 22:32:48.498586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.498567 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.522793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.522760 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7"] Oct 29 22:32:48.597006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.596968 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597019 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597184 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597215 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597245 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597297 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597442 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597512 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.597574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.597560 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.647192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.647131 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm"] Oct 29 22:32:48.651640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.651624 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.675050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.675027 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm"] Oct 29 22:32:48.698587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.698560 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699179 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699385 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699432 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699510 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699572 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699617 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699120 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699804 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.699866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699851 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.701041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.699957 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.701041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.700190 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.701041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.700233 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.701041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.700985 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.702735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.702665 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.703095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.703073 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.720192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.720169 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.720492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.720470 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.801058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.801028 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmxj2\" (UniqueName: \"kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.801195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.801091 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.801195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.801158 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.836747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.836687 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:48.903114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.903084 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmxj2\" (UniqueName: \"kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.903232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.903178 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.903232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.903225 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.904096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.904070 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.905661 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.905637 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.924105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.924083 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmxj2\" (UniqueName: \"kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2\") pod \"maas-api-revoke-sub-412b45ba-blffm\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:48.965537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:48.965511 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:49.089179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.089141 5367 generic.go:356] "Generic (PLEG): container finished" podID="49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" containerID="ae1aa5b6773aa3a07d0f4f1da4e375657a8df3c5d5ae10c7e8bc9fa723808a51" exitCode=0 Oct 29 22:32:49.089337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.089218 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" event={"ID":"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27","Type":"ContainerDied","Data":"ae1aa5b6773aa3a07d0f4f1da4e375657a8df3c5d5ae10c7e8bc9fa723808a51"} Oct 29 22:32:49.153415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.153342 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc"] Oct 29 22:32:49.161124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.161003 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.162536 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:49.162504 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746b797f_764a_4e4e_ab9a_8c61dea134ee.slice/crio-b48d7dad9e532029c3e1130afbb07bd1d31262e6d926dc43cbb799c4059c7795 WatchSource:0}: Error finding container b48d7dad9e532029c3e1130afbb07bd1d31262e6d926dc43cbb799c4059c7795: Status 404 returned error can't find the container with id b48d7dad9e532029c3e1130afbb07bd1d31262e6d926dc43cbb799c4059c7795 Oct 29 22:32:49.164113 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.164089 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7"] Oct 29 22:32:49.165333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.165295 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:49.165423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.165375 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:49.165478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.165418 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:49.202932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.202901 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc"] Oct 29 22:32:49.284018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.283990 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm"] Oct 29 22:32:49.284148 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:49.284060 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c7762e_2cae_4ae4_8eb1_35368d66f02b.slice/crio-26e6302dba15074558f24a28c7fe0baa5e5d5b0dad12c55d820ba9f3369e4a6d WatchSource:0}: Error finding container 26e6302dba15074558f24a28c7fe0baa5e5d5b0dad12c55d820ba9f3369e4a6d: Status 404 returned error can't find the container with id 26e6302dba15074558f24a28c7fe0baa5e5d5b0dad12c55d820ba9f3369e4a6d Oct 29 22:32:49.310608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.310583 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.310771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.310621 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.310898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.310860 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgp77\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.310982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.310967 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.311096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.311080 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.311393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.311265 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.311393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.311373 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.311507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.311408 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.311507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.311441 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.412886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.412802 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.412915 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413073 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413134 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413181 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413224 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413234 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413316 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413354 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413407 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413430 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgp77\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413616 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413650 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.413961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.413876 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.415833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.415809 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.415944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.415911 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.437637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.437614 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.447496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.447473 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgp77\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.498079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.497999 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:49.830177 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:32:49.830145 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44780614_e050_4e18_885d_52a560c5deb7.slice/crio-19c61ae6544c18cdff76c2fee9e8125f1fdfd0cebba5747f4e2c9d0f764aba70 WatchSource:0}: Error finding container 19c61ae6544c18cdff76c2fee9e8125f1fdfd0cebba5747f4e2c9d0f764aba70: Status 404 returned error can't find the container with id 19c61ae6544c18cdff76c2fee9e8125f1fdfd0cebba5747f4e2c9d0f764aba70 Oct 29 22:32:49.831815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.831786 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc"] Oct 29 22:32:49.832820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.832787 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:49.832925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.832852 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:49.832925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:49.832878 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:50.101864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.101822 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" event={"ID":"746b797f-764a-4e4e-ab9a-8c61dea134ee","Type":"ContainerStarted","Data":"78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936"} Oct 29 22:32:50.102601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.101894 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" event={"ID":"746b797f-764a-4e4e-ab9a-8c61dea134ee","Type":"ContainerStarted","Data":"b48d7dad9e532029c3e1130afbb07bd1d31262e6d926dc43cbb799c4059c7795"} Oct 29 22:32:50.103845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.103813 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" event={"ID":"44780614-e050-4e18-885d-52a560c5deb7","Type":"ContainerStarted","Data":"a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6"} Oct 29 22:32:50.103972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.103869 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" event={"ID":"44780614-e050-4e18-885d-52a560c5deb7","Type":"ContainerStarted","Data":"19c61ae6544c18cdff76c2fee9e8125f1fdfd0cebba5747f4e2c9d0f764aba70"} Oct 29 22:32:50.105798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.105774 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" event={"ID":"83c7762e-2cae-4ae4-8eb1-35368d66f02b","Type":"ContainerStarted","Data":"dd60f0f656fb17a2198696e8ea7a244e5a4586b404ce15292bf268f845add0a0"} Oct 29 22:32:50.105897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.105825 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" event={"ID":"83c7762e-2cae-4ae4-8eb1-35368d66f02b","Type":"ContainerStarted","Data":"26e6302dba15074558f24a28c7fe0baa5e5d5b0dad12c55d820ba9f3369e4a6d"} Oct 29 22:32:50.151775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.151146 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podStartSLOduration=2.151127617 podStartE2EDuration="2.151127617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:50.144428929 +0000 UTC m=+1037.038503848" watchObservedRunningTime="2026-10-29 22:32:50.151127617 +0000 UTC m=+1037.045202539" Oct 29 22:32:50.212651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.212590 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" podStartSLOduration=2.212569265 podStartE2EDuration="2.212569265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:50.192129892 +0000 UTC m=+1037.086204829" watchObservedRunningTime="2026-10-29 22:32:50.212569265 +0000 UTC m=+1037.106644184" Oct 29 22:32:50.242097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.242042 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podStartSLOduration=1.242024046 podStartE2EDuration="1.242024046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:50.241222089 +0000 UTC m=+1037.135297012" watchObservedRunningTime="2026-10-29 22:32:50.242024046 +0000 UTC m=+1037.136098967" Oct 29 22:32:50.316081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.316055 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:50.442179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.441558 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca\") pod \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " Oct 29 22:32:50.442179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.441670 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca\") pod \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " Oct 29 22:32:50.442179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.441949 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf\" (UniqueName: \"kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf\") pod \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\" (UID: \"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27\") " Oct 29 22:32:50.442846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.442816 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca" pod "49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" (UID: "49ceac8e-6bd2-4a5d-92b6-7a2fb900df27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:50.443252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.443226 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:50.445560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.445532 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca" pod "49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" (UID: "49ceac8e-6bd2-4a5d-92b6-7a2fb900df27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:50.445887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.445864 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf" pod "49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" (UID: "49ceac8e-6bd2-4a5d-92b6-7a2fb900df27"). InnerVolumeSpecName "kube-api-access-kplsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:50.502721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.502402 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:50.502721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.502596 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:50.502721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.502659 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:50.544669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.544633 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplsf\" (UniqueName: \"kubernetes.io/projected/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-kube-api-access-kplsf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:50.544669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.544665 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:50.837213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.837182 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:50.838292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.838272 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:32:50.838401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:50.838324 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 29 22:32:51.116212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.116129 5367 generic.go:356] "Generic (PLEG): container finished" podID="83c7762e-2cae-4ae4-8eb1-35368d66f02b" containerID="dd60f0f656fb17a2198696e8ea7a244e5a4586b404ce15292bf268f845add0a0" exitCode=0 Oct 29 22:32:51.116342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.116295 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" event={"ID":"83c7762e-2cae-4ae4-8eb1-35368d66f02b","Type":"ContainerDied","Data":"dd60f0f656fb17a2198696e8ea7a244e5a4586b404ce15292bf268f845add0a0"} Oct 29 22:32:51.117747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.117724 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" event={"ID":"49ceac8e-6bd2-4a5d-92b6-7a2fb900df27","Type":"ContainerDied","Data":"7195022d8588040ab577aa848e5ab53df8216446a9b18220af23093ba44297e0"} Oct 29 22:32:51.117798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.117767 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7195022d8588040ab577aa848e5ab53df8216446a9b18220af23093ba44297e0" Oct 29 22:32:51.118667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.118647 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6" Oct 29 22:32:51.503964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.503870 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.504234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.504202 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:51.837404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.837367 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.837794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:51.837421 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 29 22:32:52.343652 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:32:52.343618 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:32:52.437507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.437224 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:52.506821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.503740 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.506821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.503829 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:52.527095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.527043 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca\") pod \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " Oct 29 22:32:52.527495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.527462 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca" pod "83c7762e-2cae-4ae4-8eb1-35368d66f02b" (UID: "83c7762e-2cae-4ae4-8eb1-35368d66f02b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:52.527877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.527851 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2\" (UniqueName: \"kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2\") pod \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " Oct 29 22:32:52.528045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.528031 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca\") pod \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\" (UID: \"83c7762e-2cae-4ae4-8eb1-35368d66f02b\") " Oct 29 22:32:52.529125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.529101 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:52.530097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.530076 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca" pod "83c7762e-2cae-4ae4-8eb1-35368d66f02b" (UID: "83c7762e-2cae-4ae4-8eb1-35368d66f02b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:52.531919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.531899 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2" pod "83c7762e-2cae-4ae4-8eb1-35368d66f02b" (UID: "83c7762e-2cae-4ae4-8eb1-35368d66f02b"). InnerVolumeSpecName "kube-api-access-kmxj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:52.630959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.630925 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmxj2\" (UniqueName: \"kubernetes.io/projected/83c7762e-2cae-4ae4-8eb1-35368d66f02b-kube-api-access-kmxj2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:52.630959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.630952 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c7762e-2cae-4ae4-8eb1-35368d66f02b-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:32:52.837448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.837392 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.837844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:52.837479 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 29 22:32:53.135315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.135234 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" Oct 29 22:32:53.135315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.135263 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm" event={"ID":"83c7762e-2cae-4ae4-8eb1-35368d66f02b","Type":"ContainerDied","Data":"26e6302dba15074558f24a28c7fe0baa5e5d5b0dad12c55d820ba9f3369e4a6d"} Oct 29 22:32:53.135315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.135313 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26e6302dba15074558f24a28c7fe0baa5e5d5b0dad12c55d820ba9f3369e4a6d" Oct 29 22:32:53.506026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.504338 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.506026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.504401 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:53.837461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.837427 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.837858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:53.837476 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 29 22:32:54.511230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:54.511190 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.511509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:54.511480 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:54.837584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:54.837548 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.838004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:54.837596 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 29 22:32:55.498922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:55.498877 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.499128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:55.498933 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:55.837887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:55.837843 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.838358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:55.837897 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 29 22:32:56.498709 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:56.498658 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:32:56.498912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:56.498744 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 29 22:32:56.877574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:56.877539 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:56.878105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:56.878082 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:56.879231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:56.879209 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:32:57.502191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:57.502163 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:57.502659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:57.502643 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:57.503407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:57.503388 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 22:32:59.734046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.734008 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr"] Oct 29 22:32:59.735065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.735032 5367 cpu_manager.go:414] "containerMap: removing container" podUID="83c7762e-2cae-4ae4-8eb1-35368d66f02b" containerName="revoke-keys" Oct 29 22:32:59.735065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.735066 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c7762e-2cae-4ae4-8eb1-35368d66f02b" containerName="revoke-keys" Oct 29 22:32:59.735190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.735110 5367 cpu_manager.go:414] "containerMap: removing container" podUID="49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" containerName="revoke-keys" Oct 29 22:32:59.735190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.735118 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" containerName="revoke-keys" Oct 29 22:32:59.735253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.735247 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" containerName="revoke-keys" Oct 29 22:32:59.735284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.735266 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c7762e-2cae-4ae4-8eb1-35368d66f02b" containerName="revoke-keys" Oct 29 22:32:59.742177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.742156 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.756935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.756899 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr"] Oct 29 22:32:59.769739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.763869 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:59.769739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.763956 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:59.793328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.793293 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.793447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.793378 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.793582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.793560 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhnh\" (UniqueName: \"kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.834013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.833989 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:32:59.834013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.833988 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:32:59.895528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.895501 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.895640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.895551 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.895695 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.895646 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhnh\" (UniqueName: \"kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.896212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.896189 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.897930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.897908 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:32:59.915386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:32:59.915362 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhnh\" (UniqueName: \"kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh\") pod \"maas-api-revoke-sub-11cef486-nc2rr\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:33:00.059910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:00.059882 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:33:00.381901 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:00.381868 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50350920_7b34_49ab_871a_0dca19d59fab.slice/crio-07336140b826265ea67900cab31d8f25462350479d3ce6a7e28a5e9718a311e1 WatchSource:0}: Error finding container 07336140b826265ea67900cab31d8f25462350479d3ce6a7e28a5e9718a311e1: Status 404 returned error can't find the container with id 07336140b826265ea67900cab31d8f25462350479d3ce6a7e28a5e9718a311e1 Oct 29 22:33:00.382868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:00.382842 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr"] Oct 29 22:33:01.229918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:01.229876 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" event={"ID":"50350920-7b34-49ab-871a-0dca19d59fab","Type":"ContainerStarted","Data":"b1496748da4296847a5f9254a8bd6f5fa0550d0ffdd45059dcd93c7d31782b13"} Oct 29 22:33:01.230379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:01.229933 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" event={"ID":"50350920-7b34-49ab-871a-0dca19d59fab","Type":"ContainerStarted","Data":"07336140b826265ea67900cab31d8f25462350479d3ce6a7e28a5e9718a311e1"} Oct 29 22:33:01.296072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:01.296023 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" podStartSLOduration=2.2960099019999998 podStartE2EDuration="2.296009902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:01.290896283 +0000 UTC m=+1048.184971203" watchObservedRunningTime="2026-10-29 22:33:01.296009902 +0000 UTC m=+1048.190084821" Oct 29 22:33:02.013266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.013231 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7"] Oct 29 22:33:02.018119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.018093 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.036802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.036755 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7"] Oct 29 22:33:02.142460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.142415 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96vw\" (UniqueName: \"kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.142650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.142491 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.142693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.142675 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.242168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.242130 5367 generic.go:356] "Generic (PLEG): container finished" podID="50350920-7b34-49ab-871a-0dca19d59fab" containerID="b1496748da4296847a5f9254a8bd6f5fa0550d0ffdd45059dcd93c7d31782b13" exitCode=0 Oct 29 22:33:02.242783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.242210 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" event={"ID":"50350920-7b34-49ab-871a-0dca19d59fab","Type":"ContainerDied","Data":"b1496748da4296847a5f9254a8bd6f5fa0550d0ffdd45059dcd93c7d31782b13"} Oct 29 22:33:02.244069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.244043 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vw\" (UniqueName: \"kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.244212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.244078 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.244212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.244207 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.244892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.244871 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.246476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.246433 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.269223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.269144 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96vw\" (UniqueName: \"kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw\") pod \"maas-api-revoke-sub-17769705-ssmr7\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.336312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.336275 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:02.688022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:02.680355 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7"] Oct 29 22:33:03.376725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:03.369409 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" event={"ID":"00f8756e-29f9-405f-9881-fa3c62fdbdee","Type":"ContainerStarted","Data":"5486110bc2e7f354ebfd607647ba5bdaa0707be8964ed84690995706591158de"} Oct 29 22:33:04.008731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.006992 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:04.403093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.403055 5367 generic.go:356] "Generic (PLEG): container finished" podID="00f8756e-29f9-405f-9881-fa3c62fdbdee" containerID="70aba5a8871acc05ab23d52d1dbaff9f8562f3ea3c2f6dec57aa70fd6cfb6503" exitCode=0 Oct 29 22:33:04.403748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.403720 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" event={"ID":"00f8756e-29f9-405f-9881-fa3c62fdbdee","Type":"ContainerDied","Data":"70aba5a8871acc05ab23d52d1dbaff9f8562f3ea3c2f6dec57aa70fd6cfb6503"} Oct 29 22:33:04.787776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.786925 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:33:04.812922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.788038 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:04.862766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.838874 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:33:04.915723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:04.866312 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:33:05.046825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.024105 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.046825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.045313 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:33:05.347777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.347727 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.348259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.347772 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.690829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.688667 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.737410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.737380 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:33:05.770971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.770923 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:33:05.845728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.788212 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.882723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.796340 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.907724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.906780 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.908721 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.908821 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.843944 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.866808 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.866851 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.866892 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.931425 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.933985 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.957840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:05.936846 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:33:06.025629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.025591 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca\") pod \"50350920-7b34-49ab-871a-0dca19d59fab\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " Oct 29 22:33:06.075578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.075549 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh\" (UniqueName: \"kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh\") pod \"50350920-7b34-49ab-871a-0dca19d59fab\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " Oct 29 22:33:06.076238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.076215 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca\") pod \"50350920-7b34-49ab-871a-0dca19d59fab\" (UID: \"50350920-7b34-49ab-871a-0dca19d59fab\") " Oct 29 22:33:06.097159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.056141 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca" pod "50350920-7b34-49ab-871a-0dca19d59fab" (UID: "50350920-7b34-49ab-871a-0dca19d59fab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:06.099249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.037786 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:33:06.099743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.037822 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:33:06.147726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.143938 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" Oct 29 22:33:06.147726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.144149 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr" event={"ID":"50350920-7b34-49ab-871a-0dca19d59fab","Type":"ContainerDied","Data":"07336140b826265ea67900cab31d8f25462350479d3ce6a7e28a5e9718a311e1"} Oct 29 22:33:06.147726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.144191 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07336140b826265ea67900cab31d8f25462350479d3ce6a7e28a5e9718a311e1" Oct 29 22:33:06.271184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.269116 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:06.271184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.269203 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.365742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.360380 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.365742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.360894 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:33:06.426724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.385184 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh" pod "50350920-7b34-49ab-871a-0dca19d59fab" (UID: "50350920-7b34-49ab-871a-0dca19d59fab"). InnerVolumeSpecName "kube-api-access-wvhnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:06.426724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.406049 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhnh\" (UniqueName: \"kubernetes.io/projected/50350920-7b34-49ab-871a-0dca19d59fab-kube-api-access-wvhnh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:06.527774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.515957 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca" pod "50350920-7b34-49ab-871a-0dca19d59fab" (UID: "50350920-7b34-49ab-871a-0dca19d59fab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:06.549724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.538127 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50350920-7b34-49ab-871a-0dca19d59fab-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:06.880036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.878799 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.880036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.880036 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:33:06.894986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.894946 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.895516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.895487 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:06.936754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.911255 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.936754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.911553 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:06.936754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.911810 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:33:06.936754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.936270 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.936754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.936608 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:06.937189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.936870 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:33:06.937189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.937093 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:06.938978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.937418 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:33:06.938978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.937485 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:33:06.938978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:06.937521 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:33:07.204660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.168158 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.204660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.168511 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:33:07.204660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.169459 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.463835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.369119 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:33:07.879724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.879200 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.899586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.898374 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.918894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.918854 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.276726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.276664 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.276902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.959875 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:33:08.285775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.920038 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.309610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.959943 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.310941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.959965 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:08.312045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.919135 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.313019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.238825 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:33:08.317717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.314621 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.403768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.371040 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.403768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:07.959918 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:08.763724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.761849 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:08.818960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.817354 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw\" (UniqueName: \"kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw\") pod \"00f8756e-29f9-405f-9881-fa3c62fdbdee\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " Oct 29 22:33:08.818960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.817605 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca\") pod \"00f8756e-29f9-405f-9881-fa3c62fdbdee\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " Oct 29 22:33:08.818960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.817746 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca\") pod \"00f8756e-29f9-405f-9881-fa3c62fdbdee\" (UID: \"00f8756e-29f9-405f-9881-fa3c62fdbdee\") " Oct 29 22:33:08.824917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.824874 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.831919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.831887 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca" pod "00f8756e-29f9-405f-9881-fa3c62fdbdee" (UID: "00f8756e-29f9-405f-9881-fa3c62fdbdee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:08.884938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.884842 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.907729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.902825 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.962816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:08.962777 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:09.016722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.014871 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw" pod "00f8756e-29f9-405f-9881-fa3c62fdbdee" (UID: "00f8756e-29f9-405f-9881-fa3c62fdbdee"). InnerVolumeSpecName "kube-api-access-b96vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:09.017043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.017012 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca" pod "00f8756e-29f9-405f-9881-fa3c62fdbdee" (UID: "00f8756e-29f9-405f-9881-fa3c62fdbdee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:09.066182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.066142 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b96vw\" (UniqueName: \"kubernetes.io/projected/00f8756e-29f9-405f-9881-fa3c62fdbdee-kube-api-access-b96vw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:09.066439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.066411 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8756e-29f9-405f-9881-fa3c62fdbdee-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:09.253477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.239664 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.281834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.281784 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.286992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.286855 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.311737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.310981 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.311737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.311361 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.321849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.321803 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.328486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.324210 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:09.328890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.324148 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.376136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.373785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:09.376136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.375439 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.407541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.407507 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:09.407900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.407872 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.490226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.490183 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Oct 29 22:33:09.555915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.555858 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.792226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.792170 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" event={"ID":"00f8756e-29f9-405f-9881-fa3c62fdbdee","Type":"ContainerDied","Data":"5486110bc2e7f354ebfd607647ba5bdaa0707be8964ed84690995706591158de"} Oct 29 22:33:09.792508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.792470 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5486110bc2e7f354ebfd607647ba5bdaa0707be8964ed84690995706591158de" Oct 29 22:33:09.792864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.792828 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7" Oct 29 22:33:09.893965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.893862 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.908909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:09.908863 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.270816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.270688 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.286939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.286896 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:33:10.288012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.287984 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.311883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.311814 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:10.312975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.312621 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:33:10.337460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.337421 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.338107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.337685 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.338567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.337741 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:10.386669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.386620 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:10.387363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.387325 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.915791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.912181 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.956038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:10.955991 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:11.271894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.271791 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.296730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.296672 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.300628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.296939 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.315878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.315839 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.316347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.316320 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:11.343824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.343784 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:11.347880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.344062 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:33:11.348261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.344088 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:33:11.430216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.430175 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:11.430821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.430790 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:33:11.695853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.695804 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.877277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.877242 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:11.877611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.877580 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.914356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:11.914317 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:33:12.028768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.007511 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:12.028768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.017986 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.184721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.184160 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:33:12.192443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.192420 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:33:12.245999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.245969 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:33:12.249852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.249373 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:33:12.249852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.249429 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:33:12.311960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.311915 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.312711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.312658 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:33:12.322686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.322654 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.347937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.325754 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:33:12.389717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.387999 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:33:12.395720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.391385 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:33:12.395720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.392395 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:33:12.423060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.423034 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:33:12.427790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.427762 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:33:12.441580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.441557 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:33:12.441865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:12.441851 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:33:15.144364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.144331 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5"] Oct 29 22:33:15.145056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.145041 5367 cpu_manager.go:414] "containerMap: removing container" podUID="00f8756e-29f9-405f-9881-fa3c62fdbdee" containerName="revoke-keys" Oct 29 22:33:15.145100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.145058 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f8756e-29f9-405f-9881-fa3c62fdbdee" containerName="revoke-keys" Oct 29 22:33:15.145100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.145078 5367 cpu_manager.go:414] "containerMap: removing container" podUID="50350920-7b34-49ab-871a-0dca19d59fab" containerName="revoke-keys" Oct 29 22:33:15.145100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.145083 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50350920-7b34-49ab-871a-0dca19d59fab" containerName="revoke-keys" Oct 29 22:33:15.145203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.145193 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f8756e-29f9-405f-9881-fa3c62fdbdee" containerName="revoke-keys" Oct 29 22:33:15.145334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.145208 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="50350920-7b34-49ab-871a-0dca19d59fab" containerName="revoke-keys" Oct 29 22:33:15.149199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.149177 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.168874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.168838 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5"] Oct 29 22:33:15.176300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.175792 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:15.176300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.175806 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:33:15.176300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.175803 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:33:15.178233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.178216 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:15.270379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.270341 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.270559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.270428 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.270689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.270663 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.371946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.371913 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.372132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.372018 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.372132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.372074 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.372754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.372729 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.374330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.374309 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.391969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.391942 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29\") pod \"maas-api-revoke-sub-e2df52cd-bb5m5\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.463347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.463268 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:15.768865 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:15.768834 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf52d625_9d3b_4681_bb1c_8c827aaf2335.slice/crio-c59cd80fe22a16f5a7b7bfe93238b4bdfb666bc5581868305361a09e5025ec69 WatchSource:0}: Error finding container c59cd80fe22a16f5a7b7bfe93238b4bdfb666bc5581868305361a09e5025ec69: Status 404 returned error can't find the container with id c59cd80fe22a16f5a7b7bfe93238b4bdfb666bc5581868305361a09e5025ec69 Oct 29 22:33:15.770078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:15.770053 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5"] Oct 29 22:33:16.025745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.025643 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" event={"ID":"bf52d625-9d3b-4681-bb1c-8c827aaf2335","Type":"ContainerStarted","Data":"c59cd80fe22a16f5a7b7bfe93238b4bdfb666bc5581868305361a09e5025ec69"} Oct 29 22:33:16.829000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.828963 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt"] Oct 29 22:33:16.833472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.833453 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:16.853614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.853587 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt"] Oct 29 22:33:16.898193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.898157 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:16.898489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.898298 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:16.898489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:16.898380 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntx5v\" (UniqueName: \"kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.000049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.000012 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntx5v\" (UniqueName: \"kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.000217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.000195 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.000313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.000289 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.000917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.000886 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.002521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.002498 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.022400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.022372 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntx5v\" (UniqueName: \"kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v\") pod \"maas-api-revoke-sub-f1a0d7f6-gw2wt\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.038412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.038382 5367 generic.go:356] "Generic (PLEG): container finished" podID="bf52d625-9d3b-4681-bb1c-8c827aaf2335" containerID="e3e97d9fefd88c5f9661ea1bbdfd3c96fb32a3bbc5373d26089a5323917a29fa" exitCode=0 Oct 29 22:33:17.038563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.038478 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" event={"ID":"bf52d625-9d3b-4681-bb1c-8c827aaf2335","Type":"ContainerDied","Data":"e3e97d9fefd88c5f9661ea1bbdfd3c96fb32a3bbc5373d26089a5323917a29fa"} Oct 29 22:33:17.148375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.148292 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:17.455495 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:17.455464 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290d4e6c_5515_41b6_a409_c205cbe20432.slice/crio-b05c25a2f6e97c5c31bb8b7c75eb4adf699db3c214eeadf17f644638109860e8 WatchSource:0}: Error finding container b05c25a2f6e97c5c31bb8b7c75eb4adf699db3c214eeadf17f644638109860e8: Status 404 returned error can't find the container with id b05c25a2f6e97c5c31bb8b7c75eb4adf699db3c214eeadf17f644638109860e8 Oct 29 22:33:17.457270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:17.457243 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt"] Oct 29 22:33:18.055130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.055096 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" event={"ID":"290d4e6c-5515-41b6-a409-c205cbe20432","Type":"ContainerStarted","Data":"8d75e2f79be790d656f9797394ae55347db190f1749ca96cd2dac66e3f7ff8ea"} Oct 29 22:33:18.055618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.055149 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" event={"ID":"290d4e6c-5515-41b6-a409-c205cbe20432","Type":"ContainerStarted","Data":"b05c25a2f6e97c5c31bb8b7c75eb4adf699db3c214eeadf17f644638109860e8"} Oct 29 22:33:18.104254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.103784 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" podStartSLOduration=2.103759815 podStartE2EDuration="2.103759815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:18.096817211 +0000 UTC m=+1064.990892131" watchObservedRunningTime="2026-10-29 22:33:18.103759815 +0000 UTC m=+1064.997834735" Oct 29 22:33:18.239836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.239812 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:18.322273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.322192 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca\") pod \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " Oct 29 22:33:18.322450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.322371 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29\") pod \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " Oct 29 22:33:18.322590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.322572 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca\") pod \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\" (UID: \"bf52d625-9d3b-4681-bb1c-8c827aaf2335\") " Oct 29 22:33:18.322649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.322593 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca" pod "bf52d625-9d3b-4681-bb1c-8c827aaf2335" (UID: "bf52d625-9d3b-4681-bb1c-8c827aaf2335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:18.323236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.323213 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:18.324857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.324833 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29" pod "bf52d625-9d3b-4681-bb1c-8c827aaf2335" (UID: "bf52d625-9d3b-4681-bb1c-8c827aaf2335"). InnerVolumeSpecName "kube-api-access-z4v29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:18.325302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.325278 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca" pod "bf52d625-9d3b-4681-bb1c-8c827aaf2335" (UID: "bf52d625-9d3b-4681-bb1c-8c827aaf2335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:18.424417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.424390 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf52d625-9d3b-4681-bb1c-8c827aaf2335-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:18.424417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.424416 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/bf52d625-9d3b-4681-bb1c-8c827aaf2335-kube-api-access-z4v29\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:18.694756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.694636 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h"] Oct 29 22:33:18.695384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.695365 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bf52d625-9d3b-4681-bb1c-8c827aaf2335" containerName="revoke-keys" Oct 29 22:33:18.695384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.695384 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf52d625-9d3b-4681-bb1c-8c827aaf2335" containerName="revoke-keys" Oct 29 22:33:18.695527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.695518 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf52d625-9d3b-4681-bb1c-8c827aaf2335" containerName="revoke-keys" Oct 29 22:33:18.699517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.699497 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.724065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.724035 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h"] Oct 29 22:33:18.831449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831403 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkptk\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.831600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831545 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.831649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831601 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.831649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831623 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.831822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831768 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.831897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831882 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.831965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.831950 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.832046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.832030 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.832172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.832156 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934110 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934253 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934287 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkptk\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934371 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934440 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934489 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934619 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934649 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934777 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934825 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934851 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.934956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.934924 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.935219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.935199 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.935355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.935336 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.936647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.936621 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.937169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.937153 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.954932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.954873 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkptk\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:18.955037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:18.955019 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:19.017249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.017223 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:19.066267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.066241 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" Oct 29 22:33:19.066679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.066277 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5" event={"ID":"bf52d625-9d3b-4681-bb1c-8c827aaf2335","Type":"ContainerDied","Data":"c59cd80fe22a16f5a7b7bfe93238b4bdfb666bc5581868305361a09e5025ec69"} Oct 29 22:33:19.066679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.066332 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59cd80fe22a16f5a7b7bfe93238b4bdfb666bc5581868305361a09e5025ec69" Oct 29 22:33:19.071284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.071250 5367 generic.go:356] "Generic (PLEG): container finished" podID="290d4e6c-5515-41b6-a409-c205cbe20432" containerID="8d75e2f79be790d656f9797394ae55347db190f1749ca96cd2dac66e3f7ff8ea" exitCode=0 Oct 29 22:33:19.071415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.071340 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" event={"ID":"290d4e6c-5515-41b6-a409-c205cbe20432","Type":"ContainerDied","Data":"8d75e2f79be790d656f9797394ae55347db190f1749ca96cd2dac66e3f7ff8ea"} Oct 29 22:33:19.350251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.350218 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h"] Oct 29 22:33:19.354233 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:19.354208 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26754557_c763_4176_b3dc_dbd3b34a9e7d.slice/crio-616b389bd1c91efe6a00ad5b6f7a47b72327b3c471642a91df3804eb0ab63c90 WatchSource:0}: Error finding container 616b389bd1c91efe6a00ad5b6f7a47b72327b3c471642a91df3804eb0ab63c90: Status 404 returned error can't find the container with id 616b389bd1c91efe6a00ad5b6f7a47b72327b3c471642a91df3804eb0ab63c90 Oct 29 22:33:19.356991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.356953 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:33:19.357060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.357035 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:33:19.357102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:19.357074 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:33:20.091773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.091723 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" event={"ID":"26754557-c763-4176-b3dc-dbd3b34a9e7d","Type":"ContainerStarted","Data":"bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61"} Oct 29 22:33:20.092147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.091781 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" event={"ID":"26754557-c763-4176-b3dc-dbd3b34a9e7d","Type":"ContainerStarted","Data":"616b389bd1c91efe6a00ad5b6f7a47b72327b3c471642a91df3804eb0ab63c90"} Oct 29 22:33:20.147559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.147474 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podStartSLOduration=2.147456319 podStartE2EDuration="2.147456319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:20.146749561 +0000 UTC m=+1067.040824482" watchObservedRunningTime="2026-10-29 22:33:20.147456319 +0000 UTC m=+1067.041531241" Oct 29 22:33:20.299288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.299262 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:20.483422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.483335 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca\") pod \"290d4e6c-5515-41b6-a409-c205cbe20432\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " Oct 29 22:33:20.483607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.483460 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca\") pod \"290d4e6c-5515-41b6-a409-c205cbe20432\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " Oct 29 22:33:20.483607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.483511 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v\" (UniqueName: \"kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v\") pod \"290d4e6c-5515-41b6-a409-c205cbe20432\" (UID: \"290d4e6c-5515-41b6-a409-c205cbe20432\") " Oct 29 22:33:20.483769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.483732 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca" pod "290d4e6c-5515-41b6-a409-c205cbe20432" (UID: "290d4e6c-5515-41b6-a409-c205cbe20432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:20.484260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.484237 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:20.485624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.485594 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v" pod "290d4e6c-5515-41b6-a409-c205cbe20432" (UID: "290d4e6c-5515-41b6-a409-c205cbe20432"). InnerVolumeSpecName "kube-api-access-ntx5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:20.485749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.485655 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca" pod "290d4e6c-5515-41b6-a409-c205cbe20432" (UID: "290d4e6c-5515-41b6-a409-c205cbe20432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:20.586357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.586322 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d4e6c-5515-41b6-a409-c205cbe20432-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:20.586357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:20.586353 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntx5v\" (UniqueName: \"kubernetes.io/projected/290d4e6c-5515-41b6-a409-c205cbe20432-kube-api-access-ntx5v\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:21.018222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:21.018195 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:21.019463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:21.019435 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:33:21.019598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:21.019484 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:33:21.101991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:21.101965 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" Oct 29 22:33:21.102386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:21.101959 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt" event={"ID":"290d4e6c-5515-41b6-a409-c205cbe20432","Type":"ContainerDied","Data":"b05c25a2f6e97c5c31bb8b7c75eb4adf699db3c214eeadf17f644638109860e8"} Oct 29 22:33:21.102386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:21.102103 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b05c25a2f6e97c5c31bb8b7c75eb4adf699db3c214eeadf17f644638109860e8" Oct 29 22:33:22.017937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:22.017902 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:33:22.018110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:22.017954 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:33:22.334886 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:33:22.334838 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:33:23.018567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:23.018526 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:33:23.018777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:23.018595 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:33:24.018391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:24.018354 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:33:24.018840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:24.018416 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:33:25.018067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:25.018025 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:33:25.018249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:25.018094 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:33:26.017882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:26.017845 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:33:26.018339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:26.017911 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:33:27.022677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:27.022644 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:27.024933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:27.023146 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:27.024933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:27.023797 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:33:28.396690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.396656 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg"] Oct 29 22:33:28.397387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.397371 5367 cpu_manager.go:414] "containerMap: removing container" podUID="290d4e6c-5515-41b6-a409-c205cbe20432" containerName="revoke-keys" Oct 29 22:33:28.397433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.397388 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290d4e6c-5515-41b6-a409-c205cbe20432" containerName="revoke-keys" Oct 29 22:33:28.397507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.397498 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="290d4e6c-5515-41b6-a409-c205cbe20432" containerName="revoke-keys" Oct 29 22:33:28.400939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.400925 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.420620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.420591 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg"] Oct 29 22:33:28.421623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.421598 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:33:28.427976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.427953 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:28.428076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.428057 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:33:28.428156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.428138 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:28.541097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.541061 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.541287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.541132 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.541287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.541204 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.643214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.643172 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.643405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.643256 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.643405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.643301 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.644274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.644246 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.645939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.645914 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.664525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.664468 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c\") pod \"maas-api-revoke-sub-42c373c4-tc9qg\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:28.715005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:28.714977 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:29.030154 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:29.030121 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb077c7c9_cbe7_47d0_a12e_e045abfe2158.slice/crio-fc6372efbd4b4a08d6f6eecd200aff9630e84976254ed89d69b7ad984d6e7f78 WatchSource:0}: Error finding container fc6372efbd4b4a08d6f6eecd200aff9630e84976254ed89d69b7ad984d6e7f78: Status 404 returned error can't find the container with id fc6372efbd4b4a08d6f6eecd200aff9630e84976254ed89d69b7ad984d6e7f78 Oct 29 22:33:29.032343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.032286 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg"] Oct 29 22:33:29.038965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.038939 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr"] Oct 29 22:33:29.045060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.045037 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.061502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.061475 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr"] Oct 29 22:33:29.151642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.151612 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.151850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.151664 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867sx\" (UniqueName: \"kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.151850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.151787 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.204818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.204786 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" event={"ID":"b077c7c9-cbe7-47d0-a12e-e045abfe2158","Type":"ContainerStarted","Data":"fc6372efbd4b4a08d6f6eecd200aff9630e84976254ed89d69b7ad984d6e7f78"} Oct 29 22:33:29.253203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.253175 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.253343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.253228 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867sx\" (UniqueName: \"kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.253343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.253260 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.253969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.253945 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.255609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.255585 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.274003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.273984 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-867sx\" (UniqueName: \"kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx\") pod \"maas-api-revoke-sub-1ad3e8be-p5lsr\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.377939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.377912 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:29.693860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:29.693826 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr"] Oct 29 22:33:29.694383 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:29.694350 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e2d128_35b4_48b4_b53d_f9a6085556b7.slice/crio-6fb03109f4408eea9c18cc71589a0da9429eb2ca6204b3e73e0f65658cc2b74b WatchSource:0}: Error finding container 6fb03109f4408eea9c18cc71589a0da9429eb2ca6204b3e73e0f65658cc2b74b: Status 404 returned error can't find the container with id 6fb03109f4408eea9c18cc71589a0da9429eb2ca6204b3e73e0f65658cc2b74b Oct 29 22:33:30.216171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:30.216079 5367 generic.go:356] "Generic (PLEG): container finished" podID="b077c7c9-cbe7-47d0-a12e-e045abfe2158" containerID="8457c2b33e791b9e62eb55ff189e26dafb870966e853b284c2ca52498d5e4e23" exitCode=0 Oct 29 22:33:30.216335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:30.216166 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" event={"ID":"b077c7c9-cbe7-47d0-a12e-e045abfe2158","Type":"ContainerDied","Data":"8457c2b33e791b9e62eb55ff189e26dafb870966e853b284c2ca52498d5e4e23"} Oct 29 22:33:30.217978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:30.217958 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" event={"ID":"f5e2d128-35b4-48b4-b53d-f9a6085556b7","Type":"ContainerStarted","Data":"df859a437eceb0aee86757902a848437d0a39ee3c3580a4f8617131e0085fae5"} Oct 29 22:33:30.218058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:30.217998 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" event={"ID":"f5e2d128-35b4-48b4-b53d-f9a6085556b7","Type":"ContainerStarted","Data":"6fb03109f4408eea9c18cc71589a0da9429eb2ca6204b3e73e0f65658cc2b74b"} Oct 29 22:33:30.304097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:30.304046 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" podStartSLOduration=1.304031959 podStartE2EDuration="1.304031959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:30.299267735 +0000 UTC m=+1077.193342656" watchObservedRunningTime="2026-10-29 22:33:30.304031959 +0000 UTC m=+1077.198106908" Oct 29 22:33:31.228513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.228475 5367 generic.go:356] "Generic (PLEG): container finished" podID="f5e2d128-35b4-48b4-b53d-f9a6085556b7" containerID="df859a437eceb0aee86757902a848437d0a39ee3c3580a4f8617131e0085fae5" exitCode=0 Oct 29 22:33:31.228946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.228549 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" event={"ID":"f5e2d128-35b4-48b4-b53d-f9a6085556b7","Type":"ContainerDied","Data":"df859a437eceb0aee86757902a848437d0a39ee3c3580a4f8617131e0085fae5"} Oct 29 22:33:31.422905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.422879 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:31.591578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.591543 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca\") pod \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " Oct 29 22:33:31.591848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.591652 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca\") pod \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " Oct 29 22:33:31.591973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.591862 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c\") pod \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\" (UID: \"b077c7c9-cbe7-47d0-a12e-e045abfe2158\") " Oct 29 22:33:31.592823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.592793 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca" pod "b077c7c9-cbe7-47d0-a12e-e045abfe2158" (UID: "b077c7c9-cbe7-47d0-a12e-e045abfe2158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:31.594073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.594046 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c" pod "b077c7c9-cbe7-47d0-a12e-e045abfe2158" (UID: "b077c7c9-cbe7-47d0-a12e-e045abfe2158"). InnerVolumeSpecName "kube-api-access-gq56c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:31.594196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.594116 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca" pod "b077c7c9-cbe7-47d0-a12e-e045abfe2158" (UID: "b077c7c9-cbe7-47d0-a12e-e045abfe2158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:31.694377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.694344 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/b077c7c9-cbe7-47d0-a12e-e045abfe2158-kube-api-access-gq56c\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:31.694377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.694372 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:31.694377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:31.694382 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b077c7c9-cbe7-47d0-a12e-e045abfe2158-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:32.238653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.238624 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" Oct 29 22:33:32.239127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.238663 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg" event={"ID":"b077c7c9-cbe7-47d0-a12e-e045abfe2158","Type":"ContainerDied","Data":"fc6372efbd4b4a08d6f6eecd200aff9630e84976254ed89d69b7ad984d6e7f78"} Oct 29 22:33:32.239127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.238734 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6372efbd4b4a08d6f6eecd200aff9630e84976254ed89d69b7ad984d6e7f78" Oct 29 22:33:32.410652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.410625 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:32.510277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.510194 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx\" (UniqueName: \"kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx\") pod \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " Oct 29 22:33:32.510277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.510244 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca\") pod \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " Oct 29 22:33:32.510480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.510330 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca\") pod \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\" (UID: \"f5e2d128-35b4-48b4-b53d-f9a6085556b7\") " Oct 29 22:33:32.510769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.510745 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca" pod "f5e2d128-35b4-48b4-b53d-f9a6085556b7" (UID: "f5e2d128-35b4-48b4-b53d-f9a6085556b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:32.511104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.511083 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:32.512445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.512421 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca" pod "f5e2d128-35b4-48b4-b53d-f9a6085556b7" (UID: "f5e2d128-35b4-48b4-b53d-f9a6085556b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:32.512669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.512651 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx" pod "f5e2d128-35b4-48b4-b53d-f9a6085556b7" (UID: "f5e2d128-35b4-48b4-b53d-f9a6085556b7"). InnerVolumeSpecName "kube-api-access-867sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:32.612459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.612406 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867sx\" (UniqueName: \"kubernetes.io/projected/f5e2d128-35b4-48b4-b53d-f9a6085556b7-kube-api-access-867sx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:32.612459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:32.612456 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e2d128-35b4-48b4-b53d-f9a6085556b7-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:33.249916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:33.249886 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" Oct 29 22:33:33.250285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:33.249926 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr" event={"ID":"f5e2d128-35b4-48b4-b53d-f9a6085556b7","Type":"ContainerDied","Data":"6fb03109f4408eea9c18cc71589a0da9429eb2ca6204b3e73e0f65658cc2b74b"} Oct 29 22:33:33.250285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:33.249978 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb03109f4408eea9c18cc71589a0da9429eb2ca6204b3e73e0f65658cc2b74b" Oct 29 22:33:37.348362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.348313 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6"] Oct 29 22:33:37.349293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.349269 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b077c7c9-cbe7-47d0-a12e-e045abfe2158" containerName="revoke-keys" Oct 29 22:33:37.349293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.349291 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b077c7c9-cbe7-47d0-a12e-e045abfe2158" containerName="revoke-keys" Oct 29 22:33:37.349459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.349309 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f5e2d128-35b4-48b4-b53d-f9a6085556b7" containerName="revoke-keys" Oct 29 22:33:37.349459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.349316 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e2d128-35b4-48b4-b53d-f9a6085556b7" containerName="revoke-keys" Oct 29 22:33:37.349559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.349538 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b077c7c9-cbe7-47d0-a12e-e045abfe2158" containerName="revoke-keys" Oct 29 22:33:37.349559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.349553 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e2d128-35b4-48b4-b53d-f9a6085556b7" containerName="revoke-keys" Oct 29 22:33:37.354714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.354682 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.375232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.375203 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6"] Oct 29 22:33:37.380869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.380020 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:37.380869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.380102 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:33:37.380869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.380286 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:33:37.389748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.384011 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:37.389748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.387194 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.389748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.387274 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjq2\" (UniqueName: \"kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.389748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.387469 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.488783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.488750 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.488964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.488819 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjq2\" (UniqueName: \"kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.489002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.488971 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.489355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.489336 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.491218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.491199 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.509513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.509484 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjq2\" (UniqueName: \"kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2\") pod \"maas-api-revoke-sub-3b3f9ba2-tl9k6\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:37.674103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:37.673960 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:38.123223 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:38.122770 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e545ed_6331_47a0_bff1_bd549b60f753.slice/crio-a8f6fb1254c900fb541c193118c833479e6bef282df58480cdb51b659e831eb4 WatchSource:0}: Error finding container a8f6fb1254c900fb541c193118c833479e6bef282df58480cdb51b659e831eb4: Status 404 returned error can't find the container with id a8f6fb1254c900fb541c193118c833479e6bef282df58480cdb51b659e831eb4 Oct 29 22:33:38.124406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:38.124218 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6"] Oct 29 22:33:38.324244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:38.323938 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" event={"ID":"32e545ed-6331-47a0-bff1-bd549b60f753","Type":"ContainerStarted","Data":"a8f6fb1254c900fb541c193118c833479e6bef282df58480cdb51b659e831eb4"} Oct 29 22:33:39.321722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.319403 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.449144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.424912 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:39.449144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.447800 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.455900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.449433 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:33:39.469904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.457490 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:33:39.515722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.472150 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Oct 29 22:33:39.587884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.561815 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.925228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.921869 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.928727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.926450 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" event={"ID":"32e545ed-6331-47a0-bff1-bd549b60f753","Type":"ContainerStarted","Data":"d47fb550474c4b3d7b1bde746bc2fe0d555649f96f1bcd214cd21b78a507519f"} Oct 29 22:33:39.937391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.937353 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.963391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:39.963351 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.040149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.040111 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9"] Oct 29 22:33:40.061724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.046369 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.131724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.108665 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9"] Oct 29 22:33:40.288722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.288209 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.288722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.288297 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.288722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.288326 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8bz\" (UniqueName: \"kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.313723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.312182 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.322722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.319724 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:40.322722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.320784 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:33:40.322722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.322022 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.447257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.447210 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.453801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.453763 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.462914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.462829 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8bz\" (UniqueName: \"kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.472925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.463768 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:33:40.502217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.464556 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:40.502496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.464772 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.536798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.464818 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.536798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.464836 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:40.536798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.464852 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:40.631338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.631299 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:33:40.698726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.694338 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:33:40.699067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.699044 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:33:40.702829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.702807 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:33:40.922647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.922603 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.952759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.937465 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.952759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.937863 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:33:40.952759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.939371 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.952759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:40.939673 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:33:41.056724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.030804 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.346878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.346819 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.363110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.363051 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8bz\" (UniqueName: \"kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:41.392728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.366405 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-071309d2-5c4g9\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:41.471084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.471045 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:41.498901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.498850 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:33:41.611776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.531807 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:41.722247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.587115 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.726110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.587407 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.732082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.721788 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": context deadline exceeded" Oct 29 22:33:41.823724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.799008 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:41.952445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.949799 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:33:42.067681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.949863 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:42.095577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.952018 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.099724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.095821 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.099959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.051928 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.115773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:41.981914 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:42.267729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.254216 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.301897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.281257 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.301897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.282132 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zzjpn" podUID="8f18feb0-835d-45e8-a8e2-0053321ce57a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.365839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.348801 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:33:42.498723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.349879 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:33:42.498723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.349908 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:33:42.498723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.437141 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 29 22:33:42.499324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.499274 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.557719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.557191 5367 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.588225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.588170 5367 generic.go:356] "Generic (PLEG): container finished" podID="32e545ed-6331-47a0-bff1-bd549b60f753" containerID="d47fb550474c4b3d7b1bde746bc2fe0d555649f96f1bcd214cd21b78a507519f" exitCode=0 Oct 29 22:33:42.612839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.557304 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.751732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.749911 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.751732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.750009 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:42.751732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:42.750054 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:33:43.005550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.005503 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.078974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.078931 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:33:43.151724 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:33:43.114955 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.526s" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.192955 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.193011 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.193031 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" event={"ID":"32e545ed-6331-47a0-bff1-bd549b60f753","Type":"ContainerDied","Data":"d47fb550474c4b3d7b1bde746bc2fe0d555649f96f1bcd214cd21b78a507519f"} Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.193096 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.193137 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.193173 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.155617 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.155650 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:43.227778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.155667 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:33:43.308905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.308852 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.359937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.332332 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.390714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.390660 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.442810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.430548 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.528628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.528591 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.528924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.528892 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.621724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.621078 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:33:43.621724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.621175 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:33:43.621724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.621223 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:33:44.067456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:43.857845 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.067456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.051244 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.067456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.051300 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.093859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.093813 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 29 22:33:44.100723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.096822 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:33:44.101078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.101044 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.126963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.126915 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.128812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.128782 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.221838 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.221949 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.148227 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.222001 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.222057 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.222112 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.222150 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.222190 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:44.306836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.222236 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.367647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.367475 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.367647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.367531 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.799952 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:33:44.770442 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.542s" Oct 29 22:33:44.799952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:44.770492 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" event={"ID":"6f95ec48-ba67-4903-b572-1972373c3f33","Type":"ContainerStarted","Data":"888d398da730645f75288fdc933c5410d567fb70a7211b52f97c5d66e6526907"} Oct 29 22:33:45.661717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.660805 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.678089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.678038 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.696844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.696765 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.705790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.680085 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.720098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.720066 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.720275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.720260 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:33:45.720918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.720895 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:33:45.721976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.721954 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:33:45.722631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.722608 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.747063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.747038 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:33:45.747269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.747253 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:33:45.747402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.747390 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:33:45.807871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.796056 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:45.847545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.847512 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9"] Oct 29 22:33:45.857233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.857200 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:33:45.875502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.875480 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:33:45.876383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.876357 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:33:45.884962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.884943 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:45.916718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.915167 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:33:45.916718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.915546 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:33:45.923713 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.920764 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:33:45.928675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.928655 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:33:45.929173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.929148 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:33:45.930979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:45.930935 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:33:46.059291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.059257 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca\") pod \"32e545ed-6331-47a0-bff1-bd549b60f753\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " Oct 29 22:33:46.059485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.059467 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca\") pod \"32e545ed-6331-47a0-bff1-bd549b60f753\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " Oct 29 22:33:46.059680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.059662 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2\" (UniqueName: \"kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2\") pod \"32e545ed-6331-47a0-bff1-bd549b60f753\" (UID: \"32e545ed-6331-47a0-bff1-bd549b60f753\") " Oct 29 22:33:46.059922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.059890 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca" pod "32e545ed-6331-47a0-bff1-bd549b60f753" (UID: "32e545ed-6331-47a0-bff1-bd549b60f753"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:46.061894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.061869 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca" pod "32e545ed-6331-47a0-bff1-bd549b60f753" (UID: "32e545ed-6331-47a0-bff1-bd549b60f753"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:46.062053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.062035 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:46.066245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.066218 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2" pod "32e545ed-6331-47a0-bff1-bd549b60f753" (UID: "32e545ed-6331-47a0-bff1-bd549b60f753"). InnerVolumeSpecName "kube-api-access-rvjq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:46.168330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.168292 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjq2\" (UniqueName: \"kubernetes.io/projected/32e545ed-6331-47a0-bff1-bd549b60f753-kube-api-access-rvjq2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:46.168330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.168330 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e545ed-6331-47a0-bff1-bd549b60f753-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:46.719686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.719619 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" Oct 29 22:33:46.719686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.719654 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6" event={"ID":"32e545ed-6331-47a0-bff1-bd549b60f753","Type":"ContainerDied","Data":"a8f6fb1254c900fb541c193118c833479e6bef282df58480cdb51b659e831eb4"} Oct 29 22:33:46.719974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.719798 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f6fb1254c900fb541c193118c833479e6bef282df58480cdb51b659e831eb4" Oct 29 22:33:46.721770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.721745 5367 generic.go:356] "Generic (PLEG): container finished" podID="6f95ec48-ba67-4903-b572-1972373c3f33" containerID="a0b34498931a705d97262ab653e68a3766a8e44dde76b719609a0d18a29bb74d" exitCode=0 Oct 29 22:33:46.721892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.721794 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" event={"ID":"6f95ec48-ba67-4903-b572-1972373c3f33","Type":"ContainerDied","Data":"a0b34498931a705d97262ab653e68a3766a8e44dde76b719609a0d18a29bb74d"} Oct 29 22:33:46.878557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.878524 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx"] Oct 29 22:33:46.879267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.879250 5367 cpu_manager.go:414] "containerMap: removing container" podUID="32e545ed-6331-47a0-bff1-bd549b60f753" containerName="revoke-keys" Oct 29 22:33:46.879309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.879268 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e545ed-6331-47a0-bff1-bd549b60f753" containerName="revoke-keys" Oct 29 22:33:46.879387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.879379 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e545ed-6331-47a0-bff1-bd549b60f753" containerName="revoke-keys" Oct 29 22:33:46.883281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.883262 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:46.902083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.902060 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx"] Oct 29 22:33:46.979133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.979060 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:46.979133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.979131 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqj8\" (UniqueName: \"kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:46.979306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:46.979289 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.081364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.081322 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.081534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.081436 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqj8\" (UniqueName: \"kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.081674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.081635 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.082779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.082753 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.084833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.084795 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.103616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.103593 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqj8\" (UniqueName: \"kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8\") pod \"maas-api-revoke-sub-5a793618-vc5mx\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.197758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.197689 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:47.504974 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:47.504945 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609a5784_7d16_49e2_9db8_074e0d3ad8c0.slice/crio-cc243ccc3ac14b460a361d35584c06b64d98f283ee7c45b4e213651d8645b7b2 WatchSource:0}: Error finding container cc243ccc3ac14b460a361d35584c06b64d98f283ee7c45b4e213651d8645b7b2: Status 404 returned error can't find the container with id cc243ccc3ac14b460a361d35584c06b64d98f283ee7c45b4e213651d8645b7b2 Oct 29 22:33:47.506536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.506514 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx"] Oct 29 22:33:47.732856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.732821 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" event={"ID":"609a5784-7d16-49e2-9db8-074e0d3ad8c0","Type":"ContainerStarted","Data":"cc243ccc3ac14b460a361d35584c06b64d98f283ee7c45b4e213651d8645b7b2"} Oct 29 22:33:47.899403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.899360 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:47.996671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.996638 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca\") pod \"6f95ec48-ba67-4903-b572-1972373c3f33\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " Oct 29 22:33:47.996876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.996722 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz\" (UniqueName: \"kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz\") pod \"6f95ec48-ba67-4903-b572-1972373c3f33\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " Oct 29 22:33:47.996876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.996821 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca\") pod \"6f95ec48-ba67-4903-b572-1972373c3f33\" (UID: \"6f95ec48-ba67-4903-b572-1972373c3f33\") " Oct 29 22:33:47.997196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.997173 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca" pod "6f95ec48-ba67-4903-b572-1972373c3f33" (UID: "6f95ec48-ba67-4903-b572-1972373c3f33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:47.997513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.997494 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:47.999002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.998976 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca" pod "6f95ec48-ba67-4903-b572-1972373c3f33" (UID: "6f95ec48-ba67-4903-b572-1972373c3f33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:47.999236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:47.999213 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz" pod "6f95ec48-ba67-4903-b572-1972373c3f33" (UID: "6f95ec48-ba67-4903-b572-1972373c3f33"). InnerVolumeSpecName "kube-api-access-vc8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:48.099880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.099838 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f95ec48-ba67-4903-b572-1972373c3f33-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:48.099880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.099883 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc8bz\" (UniqueName: \"kubernetes.io/projected/6f95ec48-ba67-4903-b572-1972373c3f33-kube-api-access-vc8bz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:48.744217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.744133 5367 generic.go:356] "Generic (PLEG): container finished" podID="609a5784-7d16-49e2-9db8-074e0d3ad8c0" containerID="79f826af8055c367db0b0a620e23856e6c8ed7eff6f94b6c79ad0210f627e149" exitCode=0 Oct 29 22:33:48.744384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.744210 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" event={"ID":"609a5784-7d16-49e2-9db8-074e0d3ad8c0","Type":"ContainerDied","Data":"79f826af8055c367db0b0a620e23856e6c8ed7eff6f94b6c79ad0210f627e149"} Oct 29 22:33:48.745592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.745575 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" Oct 29 22:33:48.745729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.745620 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9" event={"ID":"6f95ec48-ba67-4903-b572-1972373c3f33","Type":"ContainerDied","Data":"888d398da730645f75288fdc933c5410d567fb70a7211b52f97c5d66e6526907"} Oct 29 22:33:48.745729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.745667 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888d398da730645f75288fdc933c5410d567fb70a7211b52f97c5d66e6526907" Oct 29 22:33:48.786107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.786085 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b"] Oct 29 22:33:48.787249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.787228 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6f95ec48-ba67-4903-b572-1972373c3f33" containerName="revoke-keys" Oct 29 22:33:48.787339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.787251 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f95ec48-ba67-4903-b572-1972373c3f33" containerName="revoke-keys" Oct 29 22:33:48.787414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.787403 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f95ec48-ba67-4903-b572-1972373c3f33" containerName="revoke-keys" Oct 29 22:33:48.791950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.791933 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:48.810180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.810155 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b"] Oct 29 22:33:48.914241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.914211 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:48.914623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.914355 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnsn\" (UniqueName: \"kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:48.914623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:48.914479 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.017278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.017190 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnsn\" (UniqueName: \"kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.018164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.018136 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.018552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.018530 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.019853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.019831 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.022973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.022800 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.039830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.039805 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnsn\" (UniqueName: \"kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn\") pod \"maas-api-revoke-sub-433293e7-2gc7b\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.110385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.110349 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:49.418950 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:49.418920 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0700e067_22a1_4234_bee0_b179b14d52ca.slice/crio-46d334d9a1acb7ec64c518c2b29076a4ae13452baa26cb764671c2a9e4732fc9 WatchSource:0}: Error finding container 46d334d9a1acb7ec64c518c2b29076a4ae13452baa26cb764671c2a9e4732fc9: Status 404 returned error can't find the container with id 46d334d9a1acb7ec64c518c2b29076a4ae13452baa26cb764671c2a9e4732fc9 Oct 29 22:33:49.421157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.421133 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b"] Oct 29 22:33:49.756648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.756610 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" event={"ID":"0700e067-22a1-4234-bee0-b179b14d52ca","Type":"ContainerStarted","Data":"46d334d9a1acb7ec64c518c2b29076a4ae13452baa26cb764671c2a9e4732fc9"} Oct 29 22:33:49.937226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:49.937164 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:50.035834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.035796 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca\") pod \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " Oct 29 22:33:50.036029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.035903 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8\" (UniqueName: \"kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8\") pod \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " Oct 29 22:33:50.036231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.036198 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca\") pod \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\" (UID: \"609a5784-7d16-49e2-9db8-074e0d3ad8c0\") " Oct 29 22:33:50.037494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.037464 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca" pod "609a5784-7d16-49e2-9db8-074e0d3ad8c0" (UID: "609a5784-7d16-49e2-9db8-074e0d3ad8c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:50.039956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.039927 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca" pod "609a5784-7d16-49e2-9db8-074e0d3ad8c0" (UID: "609a5784-7d16-49e2-9db8-074e0d3ad8c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:50.041309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.041280 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8" pod "609a5784-7d16-49e2-9db8-074e0d3ad8c0" (UID: "609a5784-7d16-49e2-9db8-074e0d3ad8c0"). InnerVolumeSpecName "kube-api-access-kkqj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:50.138149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.138124 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:50.138149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.138149 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609a5784-7d16-49e2-9db8-074e0d3ad8c0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:50.138328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.138158 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqj8\" (UniqueName: \"kubernetes.io/projected/609a5784-7d16-49e2-9db8-074e0d3ad8c0-kube-api-access-kkqj8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:50.768573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.768486 5367 generic.go:356] "Generic (PLEG): container finished" podID="0700e067-22a1-4234-bee0-b179b14d52ca" containerID="93dd4dbdd62dc89b483371981d761e0306c875b6fe726af8a9e35ac9ccc9c9f1" exitCode=0 Oct 29 22:33:50.768573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.768555 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" event={"ID":"0700e067-22a1-4234-bee0-b179b14d52ca","Type":"ContainerDied","Data":"93dd4dbdd62dc89b483371981d761e0306c875b6fe726af8a9e35ac9ccc9c9f1"} Oct 29 22:33:50.770272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.770249 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" event={"ID":"609a5784-7d16-49e2-9db8-074e0d3ad8c0","Type":"ContainerDied","Data":"cc243ccc3ac14b460a361d35584c06b64d98f283ee7c45b4e213651d8645b7b2"} Oct 29 22:33:50.770378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.770276 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx" Oct 29 22:33:50.770378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:50.770282 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc243ccc3ac14b460a361d35584c06b64d98f283ee7c45b4e213651d8645b7b2" Oct 29 22:33:51.953100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:51.953067 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:52.067881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.067852 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn\" (UniqueName: \"kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn\") pod \"0700e067-22a1-4234-bee0-b179b14d52ca\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " Oct 29 22:33:52.068038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.067914 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca\") pod \"0700e067-22a1-4234-bee0-b179b14d52ca\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " Oct 29 22:33:52.068166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.068137 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca\") pod \"0700e067-22a1-4234-bee0-b179b14d52ca\" (UID: \"0700e067-22a1-4234-bee0-b179b14d52ca\") " Oct 29 22:33:52.068318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.068296 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca" pod "0700e067-22a1-4234-bee0-b179b14d52ca" (UID: "0700e067-22a1-4234-bee0-b179b14d52ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:52.068812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.068792 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:52.070032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.070006 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn" pod "0700e067-22a1-4234-bee0-b179b14d52ca" (UID: "0700e067-22a1-4234-bee0-b179b14d52ca"). InnerVolumeSpecName "kube-api-access-8hnsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:52.070109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.070066 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca" pod "0700e067-22a1-4234-bee0-b179b14d52ca" (UID: "0700e067-22a1-4234-bee0-b179b14d52ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:52.170583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.170557 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0700e067-22a1-4234-bee0-b179b14d52ca-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:52.170583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.170581 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hnsn\" (UniqueName: \"kubernetes.io/projected/0700e067-22a1-4234-bee0-b179b14d52ca-kube-api-access-8hnsn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:52.331543 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:33:52.331461 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:33:52.790671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.790592 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" Oct 29 22:33:52.790850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.790654 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b" event={"ID":"0700e067-22a1-4234-bee0-b179b14d52ca","Type":"ContainerDied","Data":"46d334d9a1acb7ec64c518c2b29076a4ae13452baa26cb764671c2a9e4732fc9"} Oct 29 22:33:52.790850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:52.790743 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d334d9a1acb7ec64c518c2b29076a4ae13452baa26cb764671c2a9e4732fc9" Oct 29 22:33:53.485800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.485762 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g"] Oct 29 22:33:53.486613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.486590 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0700e067-22a1-4234-bee0-b179b14d52ca" containerName="revoke-keys" Oct 29 22:33:53.486613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.486613 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0700e067-22a1-4234-bee0-b179b14d52ca" containerName="revoke-keys" Oct 29 22:33:53.486849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.486675 5367 cpu_manager.go:414] "containerMap: removing container" podUID="609a5784-7d16-49e2-9db8-074e0d3ad8c0" containerName="revoke-keys" Oct 29 22:33:53.486849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.486683 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609a5784-7d16-49e2-9db8-074e0d3ad8c0" containerName="revoke-keys" Oct 29 22:33:53.486944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.486923 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="609a5784-7d16-49e2-9db8-074e0d3ad8c0" containerName="revoke-keys" Oct 29 22:33:53.486992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.486944 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0700e067-22a1-4234-bee0-b179b14d52ca" containerName="revoke-keys" Oct 29 22:33:53.490712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.490676 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.513161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.513135 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g"] Oct 29 22:33:53.517441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.517420 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:33:53.517616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.517497 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:33:53.517748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.517622 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:53.520444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.520424 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:53.597511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.597485 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.597655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.597533 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.597762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.597661 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzhs\" (UniqueName: \"kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.698840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.698805 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.699012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.698868 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.699012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.698973 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzhs\" (UniqueName: \"kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.699513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.699490 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.701139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.701121 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.722031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.722010 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzhs\" (UniqueName: \"kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs\") pod \"maas-api-revoke-sub-21dfdd1c-tqs8g\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:53.810667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:53.810514 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:54.116313 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:54.116281 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413807bc_aee3_474b_a26e_e1f9ac27e8fc.slice/crio-defce4bbb3fb83c57bed3e7694453f6dc59236537d38745de5c0860ee6bd58ff WatchSource:0}: Error finding container defce4bbb3fb83c57bed3e7694453f6dc59236537d38745de5c0860ee6bd58ff: Status 404 returned error can't find the container with id defce4bbb3fb83c57bed3e7694453f6dc59236537d38745de5c0860ee6bd58ff Oct 29 22:33:54.119655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:54.119627 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g"] Oct 29 22:33:54.809904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:54.809833 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" event={"ID":"413807bc-aee3-474b-a26e-e1f9ac27e8fc","Type":"ContainerStarted","Data":"024414ec47bf3f3eaa395bf4da874c4b28c929121df5afad8cb6e3d338be93ca"} Oct 29 22:33:54.809904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:54.809890 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" event={"ID":"413807bc-aee3-474b-a26e-e1f9ac27e8fc","Type":"ContainerStarted","Data":"defce4bbb3fb83c57bed3e7694453f6dc59236537d38745de5c0860ee6bd58ff"} Oct 29 22:33:54.856624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:54.856566 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" podStartSLOduration=1.8565492730000002 podStartE2EDuration="1.856549273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:54.855612218 +0000 UTC m=+1101.749687134" watchObservedRunningTime="2026-10-29 22:33:54.856549273 +0000 UTC m=+1101.750624193" Oct 29 22:33:55.820590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:55.820561 5367 generic.go:356] "Generic (PLEG): container finished" podID="413807bc-aee3-474b-a26e-e1f9ac27e8fc" containerID="024414ec47bf3f3eaa395bf4da874c4b28c929121df5afad8cb6e3d338be93ca" exitCode=0 Oct 29 22:33:55.821015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:55.820646 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" event={"ID":"413807bc-aee3-474b-a26e-e1f9ac27e8fc","Type":"ContainerDied","Data":"024414ec47bf3f3eaa395bf4da874c4b28c929121df5afad8cb6e3d338be93ca"} Oct 29 22:33:57.051204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.051161 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:57.192645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.192557 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca\") pod \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " Oct 29 22:33:57.192827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.192808 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs\" (UniqueName: \"kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs\") pod \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " Oct 29 22:33:57.192908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.192875 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca\") pod \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\" (UID: \"413807bc-aee3-474b-a26e-e1f9ac27e8fc\") " Oct 29 22:33:57.193001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.192959 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca" pod "413807bc-aee3-474b-a26e-e1f9ac27e8fc" (UID: "413807bc-aee3-474b-a26e-e1f9ac27e8fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:57.193924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.193899 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:57.195292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.195268 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs" pod "413807bc-aee3-474b-a26e-e1f9ac27e8fc" (UID: "413807bc-aee3-474b-a26e-e1f9ac27e8fc"). InnerVolumeSpecName "kube-api-access-bkzhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:57.195378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.195327 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca" pod "413807bc-aee3-474b-a26e-e1f9ac27e8fc" (UID: "413807bc-aee3-474b-a26e-e1f9ac27e8fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:57.295227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.295195 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzhs\" (UniqueName: \"kubernetes.io/projected/413807bc-aee3-474b-a26e-e1f9ac27e8fc-kube-api-access-bkzhs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:57.295227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.295224 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413807bc-aee3-474b-a26e-e1f9ac27e8fc-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:33:57.840375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.840348 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" Oct 29 22:33:57.840550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.840384 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g" event={"ID":"413807bc-aee3-474b-a26e-e1f9ac27e8fc","Type":"ContainerDied","Data":"defce4bbb3fb83c57bed3e7694453f6dc59236537d38745de5c0860ee6bd58ff"} Oct 29 22:33:57.840550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:57.840455 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="defce4bbb3fb83c57bed3e7694453f6dc59236537d38745de5c0860ee6bd58ff" Oct 29 22:33:59.154256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.154223 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49"] Oct 29 22:33:59.154973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.154957 5367 cpu_manager.go:414] "containerMap: removing container" podUID="413807bc-aee3-474b-a26e-e1f9ac27e8fc" containerName="revoke-keys" Oct 29 22:33:59.154973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.154974 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413807bc-aee3-474b-a26e-e1f9ac27e8fc" containerName="revoke-keys" Oct 29 22:33:59.155085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.155075 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="413807bc-aee3-474b-a26e-e1f9ac27e8fc" containerName="revoke-keys" Oct 29 22:33:59.159058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.159039 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.180496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.180471 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49"] Oct 29 22:33:59.186444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.186419 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:59.186602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.186474 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:59.186602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.186531 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:33:59.186955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.186927 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:33:59.223538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.223506 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.223791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.223612 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vt4\" (UniqueName: \"kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.223791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.223718 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.325840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.325805 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.326047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.326030 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.326089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.326078 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vt4\" (UniqueName: \"kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.326792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.326761 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.328335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.328311 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.346475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.346430 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vt4\" (UniqueName: \"kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4\") pod \"maas-api-revoke-sub-eceb64c6-thm49\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.473977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.473901 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:33:59.787641 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:33:59.787614 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf757d63e_02cf_4db6_828d_59ed1c554833.slice/crio-b372ea69e6045a364c73e0dcf459b8d8273c3e4f22240a701157d2b775df4231 WatchSource:0}: Error finding container b372ea69e6045a364c73e0dcf459b8d8273c3e4f22240a701157d2b775df4231: Status 404 returned error can't find the container with id b372ea69e6045a364c73e0dcf459b8d8273c3e4f22240a701157d2b775df4231 Oct 29 22:33:59.790994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.790962 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49"] Oct 29 22:33:59.860212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:33:59.860182 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" event={"ID":"f757d63e-02cf-4db6-828d-59ed1c554833","Type":"ContainerStarted","Data":"b372ea69e6045a364c73e0dcf459b8d8273c3e4f22240a701157d2b775df4231"} Oct 29 22:34:00.871924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:00.871889 5367 generic.go:356] "Generic (PLEG): container finished" podID="f757d63e-02cf-4db6-828d-59ed1c554833" containerID="e1a255e5aa4d4853a7604eea2759a130d504ccc2fadb2fb106e80c1adfd70c4a" exitCode=0 Oct 29 22:34:00.872301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:00.871971 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" event={"ID":"f757d63e-02cf-4db6-828d-59ed1c554833","Type":"ContainerDied","Data":"e1a255e5aa4d4853a7604eea2759a130d504ccc2fadb2fb106e80c1adfd70c4a"} Oct 29 22:34:01.267650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.267610 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj"] Oct 29 22:34:01.272434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.272408 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.292586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.292558 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj"] Oct 29 22:34:01.356461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.356426 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rhv\" (UniqueName: \"kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.356654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.356471 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.356654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.356623 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.458575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.458537 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rhv\" (UniqueName: \"kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.458575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.458577 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.458802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.458620 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.459272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.459243 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.460966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.460943 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.488157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.488120 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rhv\" (UniqueName: \"kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv\") pod \"maas-api-revoke-sub-f3a85481-fcclj\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.586663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.586628 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:01.896645 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:01.896610 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb07162_8117_40c0_92dd_d0b4e8ef54ec.slice/crio-fd082d1780dc36d7a16359e2e4d6b3be70dc93cc87444539a3bf228c15bea2b6 WatchSource:0}: Error finding container fd082d1780dc36d7a16359e2e4d6b3be70dc93cc87444539a3bf228c15bea2b6: Status 404 returned error can't find the container with id fd082d1780dc36d7a16359e2e4d6b3be70dc93cc87444539a3bf228c15bea2b6 Oct 29 22:34:01.900733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:01.900637 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj"] Oct 29 22:34:02.039551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.039522 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:34:02.070067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.070039 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4\" (UniqueName: \"kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4\") pod \"f757d63e-02cf-4db6-828d-59ed1c554833\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " Oct 29 22:34:02.070222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.070124 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca\") pod \"f757d63e-02cf-4db6-828d-59ed1c554833\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " Oct 29 22:34:02.070222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.070197 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca\") pod \"f757d63e-02cf-4db6-828d-59ed1c554833\" (UID: \"f757d63e-02cf-4db6-828d-59ed1c554833\") " Oct 29 22:34:02.070553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.070529 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca" pod "f757d63e-02cf-4db6-828d-59ed1c554833" (UID: "f757d63e-02cf-4db6-828d-59ed1c554833"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:02.070986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.070964 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:02.072229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.072205 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca" pod "f757d63e-02cf-4db6-828d-59ed1c554833" (UID: "f757d63e-02cf-4db6-828d-59ed1c554833"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:02.072307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.072240 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4" pod "f757d63e-02cf-4db6-828d-59ed1c554833" (UID: "f757d63e-02cf-4db6-828d-59ed1c554833"). InnerVolumeSpecName "kube-api-access-p6vt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:02.180504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.175784 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757d63e-02cf-4db6-828d-59ed1c554833-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:02.180504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.175820 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vt4\" (UniqueName: \"kubernetes.io/projected/f757d63e-02cf-4db6-828d-59ed1c554833-kube-api-access-p6vt4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:02.892998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.892964 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" event={"ID":"acb07162-8117-40c0-92dd-d0b4e8ef54ec","Type":"ContainerStarted","Data":"589979fee162eb7dd4e30090897a9aabcf61dbffecd672ae9780b543aec3ab6a"} Oct 29 22:34:02.893159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.893023 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" event={"ID":"acb07162-8117-40c0-92dd-d0b4e8ef54ec","Type":"ContainerStarted","Data":"fd082d1780dc36d7a16359e2e4d6b3be70dc93cc87444539a3bf228c15bea2b6"} Oct 29 22:34:02.894568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.894542 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" Oct 29 22:34:02.894684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.894587 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49" event={"ID":"f757d63e-02cf-4db6-828d-59ed1c554833","Type":"ContainerDied","Data":"b372ea69e6045a364c73e0dcf459b8d8273c3e4f22240a701157d2b775df4231"} Oct 29 22:34:02.894684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:02.894640 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b372ea69e6045a364c73e0dcf459b8d8273c3e4f22240a701157d2b775df4231" Oct 29 22:34:03.807632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.807591 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms"] Oct 29 22:34:03.808360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.808343 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f757d63e-02cf-4db6-828d-59ed1c554833" containerName="revoke-keys" Oct 29 22:34:03.808429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.808360 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f757d63e-02cf-4db6-828d-59ed1c554833" containerName="revoke-keys" Oct 29 22:34:03.808486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.808477 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f757d63e-02cf-4db6-828d-59ed1c554833" containerName="revoke-keys" Oct 29 22:34:03.812389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.812371 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:03.835762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.835731 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms"] Oct 29 22:34:03.909531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.909495 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:03.909765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.909745 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qkc\" (UniqueName: \"kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:03.909957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.909939 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:03.910482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.910462 5367 generic.go:356] "Generic (PLEG): container finished" podID="acb07162-8117-40c0-92dd-d0b4e8ef54ec" containerID="589979fee162eb7dd4e30090897a9aabcf61dbffecd672ae9780b543aec3ab6a" exitCode=0 Oct 29 22:34:03.910561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:03.910541 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" event={"ID":"acb07162-8117-40c0-92dd-d0b4e8ef54ec","Type":"ContainerDied","Data":"589979fee162eb7dd4e30090897a9aabcf61dbffecd672ae9780b543aec3ab6a"} Oct 29 22:34:04.015852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.011842 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qkc\" (UniqueName: \"kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.015852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.012033 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.015852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.012181 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.015852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.013070 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.022584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.022551 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.039581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.038193 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qkc\" (UniqueName: \"kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc\") pod \"maas-api-revoke-sub-8948a6f8-f97ms\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.121802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.119948 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:04.137200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.130072 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:04.222284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.220835 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca\") pod \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " Oct 29 22:34:04.222284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.220989 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca\") pod \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " Oct 29 22:34:04.222284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.221041 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv\" (UniqueName: \"kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv\") pod \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\" (UID: \"acb07162-8117-40c0-92dd-d0b4e8ef54ec\") " Oct 29 22:34:04.223960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.223929 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca" pod "acb07162-8117-40c0-92dd-d0b4e8ef54ec" (UID: "acb07162-8117-40c0-92dd-d0b4e8ef54ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:04.224855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.224826 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:04.231077 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.230979 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca" pod "acb07162-8117-40c0-92dd-d0b4e8ef54ec" (UID: "acb07162-8117-40c0-92dd-d0b4e8ef54ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:04.274259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.274201 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv" pod "acb07162-8117-40c0-92dd-d0b4e8ef54ec" (UID: "acb07162-8117-40c0-92dd-d0b4e8ef54ec"). InnerVolumeSpecName "kube-api-access-m9rhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:04.328333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.327387 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb07162-8117-40c0-92dd-d0b4e8ef54ec-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:04.328333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.327420 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rhv\" (UniqueName: \"kubernetes.io/projected/acb07162-8117-40c0-92dd-d0b4e8ef54ec-kube-api-access-m9rhv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:04.544083 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:04.544055 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc29e821_2074_4aad_8644_48c4d68a15dd.slice/crio-18ff708aaf3aafcb0c3874fc1e4360b6ab9d1a7e07b7efc6ba5bc3a4f0bee69e WatchSource:0}: Error finding container 18ff708aaf3aafcb0c3874fc1e4360b6ab9d1a7e07b7efc6ba5bc3a4f0bee69e: Status 404 returned error can't find the container with id 18ff708aaf3aafcb0c3874fc1e4360b6ab9d1a7e07b7efc6ba5bc3a4f0bee69e Oct 29 22:34:04.545901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.545880 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms"] Oct 29 22:34:04.928953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.925202 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" event={"ID":"fc29e821-2074-4aad-8644-48c4d68a15dd","Type":"ContainerStarted","Data":"18ff708aaf3aafcb0c3874fc1e4360b6ab9d1a7e07b7efc6ba5bc3a4f0bee69e"} Oct 29 22:34:04.928953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.928020 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" event={"ID":"acb07162-8117-40c0-92dd-d0b4e8ef54ec","Type":"ContainerDied","Data":"fd082d1780dc36d7a16359e2e4d6b3be70dc93cc87444539a3bf228c15bea2b6"} Oct 29 22:34:04.928953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.928056 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj" Oct 29 22:34:04.928953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:04.928087 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd082d1780dc36d7a16359e2e4d6b3be70dc93cc87444539a3bf228c15bea2b6" Oct 29 22:34:05.942684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:05.942646 5367 generic.go:356] "Generic (PLEG): container finished" podID="fc29e821-2074-4aad-8644-48c4d68a15dd" containerID="51fd779e50b5ce36ce377ee427d2c398aaf44e6cc7b02b10596cfa4e992abdcf" exitCode=0 Oct 29 22:34:05.943311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:05.942864 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" event={"ID":"fc29e821-2074-4aad-8644-48c4d68a15dd","Type":"ContainerDied","Data":"51fd779e50b5ce36ce377ee427d2c398aaf44e6cc7b02b10596cfa4e992abdcf"} Oct 29 22:34:07.312253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.311776 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:07.394143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.393210 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca\") pod \"fc29e821-2074-4aad-8644-48c4d68a15dd\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " Oct 29 22:34:07.394143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.393262 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc\" (UniqueName: \"kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc\") pod \"fc29e821-2074-4aad-8644-48c4d68a15dd\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " Oct 29 22:34:07.394143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.393447 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca\") pod \"fc29e821-2074-4aad-8644-48c4d68a15dd\" (UID: \"fc29e821-2074-4aad-8644-48c4d68a15dd\") " Oct 29 22:34:07.395801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.395771 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca" pod "fc29e821-2074-4aad-8644-48c4d68a15dd" (UID: "fc29e821-2074-4aad-8644-48c4d68a15dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:07.401449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.400894 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc" pod "fc29e821-2074-4aad-8644-48c4d68a15dd" (UID: "fc29e821-2074-4aad-8644-48c4d68a15dd"). InnerVolumeSpecName "kube-api-access-t4qkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:07.402472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.401722 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca" pod "fc29e821-2074-4aad-8644-48c4d68a15dd" (UID: "fc29e821-2074-4aad-8644-48c4d68a15dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:07.496257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.496169 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:07.496472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.496452 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc29e821-2074-4aad-8644-48c4d68a15dd-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:07.496571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:07.496559 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qkc\" (UniqueName: \"kubernetes.io/projected/fc29e821-2074-4aad-8644-48c4d68a15dd-kube-api-access-t4qkc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:08.267355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:08.267302 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" event={"ID":"fc29e821-2074-4aad-8644-48c4d68a15dd","Type":"ContainerDied","Data":"18ff708aaf3aafcb0c3874fc1e4360b6ab9d1a7e07b7efc6ba5bc3a4f0bee69e"} Oct 29 22:34:08.267630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:08.267611 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ff708aaf3aafcb0c3874fc1e4360b6ab9d1a7e07b7efc6ba5bc3a4f0bee69e" Oct 29 22:34:08.267848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:08.267833 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms" Oct 29 22:34:09.106722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.105850 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:34:09.199732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.198791 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.199732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.199204 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:09.232723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.227787 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.232723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.229301 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.439782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.438945 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.488723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.479196 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.497130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.489564 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.543842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.543793 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.685721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.650968 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.776247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:09.776154 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.104786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.070190 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:34:10.122723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.111685 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:34:10.149726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.123960 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:34:10.199808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.152047 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:34:10.235723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.234993 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:34:10.235723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.235315 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:34:10.270769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.236323 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.270769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.236586 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:34:10.270769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.237339 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.270769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.237509 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.270769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.237678 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:34:10.270769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.237742 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:34:10.494143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.494041 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.495780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.495756 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:34:10.670366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.531336 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.700007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.699973 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:34:10.711720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.711439 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:34:10.711910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.711727 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:34:10.874725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.838153 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.896935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.896901 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:34:10.986910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:10.986853 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:34:11.213255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.112183 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.213255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.121966 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:34:11.213255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.204996 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:34:11.358833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.353459 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:34:11.358833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.353867 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:34:11.397723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.389821 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:34:11.414732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.398358 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:11.436742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.415410 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:11.595728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.584392 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:34:11.623735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.622906 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.710723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.706796 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.711018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.706786 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:34:11.730852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:11.723822 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:34:12.188181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.188130 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.203729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.195354 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.303045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.302999 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.303635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.268464 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 29 22:34:12.327136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.327096 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.328275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.328246 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.328474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.328447 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.455832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.426877 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:34:12.484743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.459110 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.484743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.459323 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.484743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.459459 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.484743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.459588 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:34:12.507986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.507919 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.721775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.721388 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:34:12.732038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.731992 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.736167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.736126 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:34:12.767535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.767482 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:12.770178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.770128 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:34:12.811737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.791532 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:34:12.937725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:12.906757 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:34:13.221724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.220723 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.221935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.221833 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:13.221935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.221885 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.518783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.478292 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 29 22:34:13.538427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.482098 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:34:13.722721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.722309 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.741128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.739017 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:34:13.741714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.741648 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.746718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.742120 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.746718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.742197 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.746718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.742430 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.766613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.766166 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.768934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.768551 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.769836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.769489 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:34:13.772093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.770758 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.811115 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.822731 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl"] Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.823774 5367 cpu_manager.go:414] "containerMap: removing container" podUID="acb07162-8117-40c0-92dd-d0b4e8ef54ec" containerName="revoke-keys" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.823803 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb07162-8117-40c0-92dd-d0b4e8ef54ec" containerName="revoke-keys" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.823848 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fc29e821-2074-4aad-8644-48c4d68a15dd" containerName="revoke-keys" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.823856 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc29e821-2074-4aad-8644-48c4d68a15dd" containerName="revoke-keys" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.824049 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc29e821-2074-4aad-8644-48c4d68a15dd" containerName="revoke-keys" Oct 29 22:34:13.828021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.824075 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb07162-8117-40c0-92dd-d0b4e8ef54ec" containerName="revoke-keys" Oct 29 22:34:13.833777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.829209 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.841663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.841598 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl"] Oct 29 22:34:13.847251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.847165 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:13.874040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.872938 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:34:13.874990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.874603 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:34:13.874990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.874741 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:34:13.875173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.875144 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:34:13.898865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.898839 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:13.899058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.899032 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.924731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.924667 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.945191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.945159 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:13.945422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.945391 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.949023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.948995 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmqz\" (UniqueName: \"kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:13.949316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.949295 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:13.949461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.949441 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:13.949593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.949571 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:34:13.960917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:13.950506 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:34:14.013336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.012473 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:34:14.036519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.036393 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:34:14.088905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.086490 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:34:14.109846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.109815 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:34:14.115251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.115228 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:34:14.116230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.116206 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:34:14.128224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.126644 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmqz\" (UniqueName: \"kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.129448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.128961 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:34:14.130851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.130127 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.130851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.130176 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.138796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.135604 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:34:14.143630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.141071 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.146166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.146143 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.153890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.153868 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmqz\" (UniqueName: \"kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz\") pod \"maas-api-revoke-sub-f0a17f80-2zhzl\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.184387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.184358 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:14.497417 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:14.497383 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d3de3a_89ef_4e75_8cf6_5ffffd1115a4.slice/crio-bc30ecce4050e0c52c4ac4e1d2eafa3674d1da36e78c0c6b4652639b114fa0a0 WatchSource:0}: Error finding container bc30ecce4050e0c52c4ac4e1d2eafa3674d1da36e78c0c6b4652639b114fa0a0: Status 404 returned error can't find the container with id bc30ecce4050e0c52c4ac4e1d2eafa3674d1da36e78c0c6b4652639b114fa0a0 Oct 29 22:34:14.500973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.500948 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl"] Oct 29 22:34:14.523982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:14.523954 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" event={"ID":"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4","Type":"ContainerStarted","Data":"bc30ecce4050e0c52c4ac4e1d2eafa3674d1da36e78c0c6b4652639b114fa0a0"} Oct 29 22:34:15.534630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:15.534594 5367 generic.go:356] "Generic (PLEG): container finished" podID="b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" containerID="1b370dbe39f2cf2a21584f9a6b27c7ae604d5b63d7641232003d8e2698c974e1" exitCode=0 Oct 29 22:34:15.535062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:15.534645 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" event={"ID":"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4","Type":"ContainerDied","Data":"1b370dbe39f2cf2a21584f9a6b27c7ae604d5b63d7641232003d8e2698c974e1"} Oct 29 22:34:16.709498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.709464 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:16.771515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.771478 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca\") pod \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " Oct 29 22:34:16.772479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.772446 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca\") pod \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " Oct 29 22:34:16.773054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.772148 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca" pod "b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" (UID: "b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:16.773296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.773278 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz\" (UniqueName: \"kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz\") pod \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\" (UID: \"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4\") " Oct 29 22:34:16.776304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.776281 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:16.786194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.786138 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca" pod "b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" (UID: "b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:16.789218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.789165 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz" pod "b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" (UID: "b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4"). InnerVolumeSpecName "kube-api-access-zrmqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:16.878059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.878024 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:16.878059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:16.878057 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmqz\" (UniqueName: \"kubernetes.io/projected/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4-kube-api-access-zrmqz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:17.556634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:17.556610 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" Oct 29 22:34:17.556798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:17.556636 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl" event={"ID":"b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4","Type":"ContainerDied","Data":"bc30ecce4050e0c52c4ac4e1d2eafa3674d1da36e78c0c6b4652639b114fa0a0"} Oct 29 22:34:17.556798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:17.556682 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc30ecce4050e0c52c4ac4e1d2eafa3674d1da36e78c0c6b4652639b114fa0a0" Oct 29 22:34:21.195419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.195380 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5"] Oct 29 22:34:21.196148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.196131 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" containerName="revoke-keys" Oct 29 22:34:21.196190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.196149 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" containerName="revoke-keys" Oct 29 22:34:21.196277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.196268 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" containerName="revoke-keys" Oct 29 22:34:21.200033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.200014 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.215813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.215785 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:34:21.225367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.225340 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5"] Oct 29 22:34:21.226095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.226077 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:34:21.226366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.226344 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:34:21.226366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.226342 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:34:21.266622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.266588 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.266809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.266643 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjsdx\" (UniqueName: \"kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.267021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.266996 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.368338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.368300 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.368494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.368458 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.368570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.368504 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjsdx\" (UniqueName: \"kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.369140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.369111 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.370798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.370774 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.390081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.390051 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjsdx\" (UniqueName: \"kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx\") pod \"maas-api-revoke-sub-aa2c9ba9-7rvm5\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.514524 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.514419 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:21.833183 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:21.833148 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb274dc39_92a7_4a00_894f_57b6314671c4.slice/crio-0b635d0ffc39034572b2a70f1fb6983e9660f077435fbc136776f2085ccd7173 WatchSource:0}: Error finding container 0b635d0ffc39034572b2a70f1fb6983e9660f077435fbc136776f2085ccd7173: Status 404 returned error can't find the container with id 0b635d0ffc39034572b2a70f1fb6983e9660f077435fbc136776f2085ccd7173 Oct 29 22:34:21.834355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:21.834331 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5"] Oct 29 22:34:22.332571 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:34:22.332536 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:34:22.668554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:22.668519 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" event={"ID":"b274dc39-92a7-4a00-894f-57b6314671c4","Type":"ContainerStarted","Data":"c681bd642c66e9441817abff5ffbb6bff15703d0300ac593cdb24a40985db449"} Oct 29 22:34:22.668831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:22.668585 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" event={"ID":"b274dc39-92a7-4a00-894f-57b6314671c4","Type":"ContainerStarted","Data":"0b635d0ffc39034572b2a70f1fb6983e9660f077435fbc136776f2085ccd7173"} Oct 29 22:34:22.711245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:22.711193 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" podStartSLOduration=1.711178604 podStartE2EDuration="1.711178604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:22.709212585 +0000 UTC m=+1129.603287521" watchObservedRunningTime="2026-10-29 22:34:22.711178604 +0000 UTC m=+1129.605253558" Oct 29 22:34:23.682780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:23.682736 5367 generic.go:356] "Generic (PLEG): container finished" podID="b274dc39-92a7-4a00-894f-57b6314671c4" containerID="c681bd642c66e9441817abff5ffbb6bff15703d0300ac593cdb24a40985db449" exitCode=0 Oct 29 22:34:23.683223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:23.682819 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" event={"ID":"b274dc39-92a7-4a00-894f-57b6314671c4","Type":"ContainerDied","Data":"c681bd642c66e9441817abff5ffbb6bff15703d0300ac593cdb24a40985db449"} Oct 29 22:34:24.130594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.130558 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5"] Oct 29 22:34:24.135294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.135274 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.151723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.150923 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f922c6e9\"" Oct 29 22:34:24.151723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.151225 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f922c6e9-dockercfg-27j8w\"" Oct 29 22:34:24.157757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.157731 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5"] Oct 29 22:34:24.303636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.303233 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvn66\" (UniqueName: \"kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66\") pod \"payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.303636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.303322 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.350744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.348459 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s"] Oct 29 22:34:24.380799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.364940 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.415731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.415401 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvn66\" (UniqueName: \"kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66\") pod \"payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.415731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.415480 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.415731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.415626 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume\") pod \"payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.418743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.418193 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s"] Oct 29 22:34:24.418743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.418278 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.418743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.418352 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqhz\" (UniqueName: \"kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz\") pod \"payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.438217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.438185 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvn66\" (UniqueName: \"kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66\") pod \"payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.450493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.450469 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:24.523028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.522197 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume\") pod \"payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.523028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.522485 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqhz\" (UniqueName: \"kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz\") pod \"payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.523384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.523223 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume\") pod \"payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.547057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.546339 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqhz\" (UniqueName: \"kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz\") pod \"payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.721932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.721894 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:24.800357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.800306 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5"] Oct 29 22:34:24.922915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:24.922883 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:25.034718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.034632 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx\" (UniqueName: \"kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx\") pod \"b274dc39-92a7-4a00-894f-57b6314671c4\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " Oct 29 22:34:25.034718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.034692 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca\") pod \"b274dc39-92a7-4a00-894f-57b6314671c4\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " Oct 29 22:34:25.034980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.034962 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca\") pod \"b274dc39-92a7-4a00-894f-57b6314671c4\" (UID: \"b274dc39-92a7-4a00-894f-57b6314671c4\") " Oct 29 22:34:25.035268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.035246 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca" pod "b274dc39-92a7-4a00-894f-57b6314671c4" (UID: "b274dc39-92a7-4a00-894f-57b6314671c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:25.035607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.035585 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:25.037083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.037058 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca" pod "b274dc39-92a7-4a00-894f-57b6314671c4" (UID: "b274dc39-92a7-4a00-894f-57b6314671c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:25.037453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.037432 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx" pod "b274dc39-92a7-4a00-894f-57b6314671c4" (UID: "b274dc39-92a7-4a00-894f-57b6314671c4"). InnerVolumeSpecName "kube-api-access-cjsdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:25.074968 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:25.074943 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b37d2d_9fb0_4ff3_9ce4_f4d64a21c7cb.slice/crio-92220d90a71377bcd5d827a15b9fed7ba3d9763763331de829b01fa96e2a6b96 WatchSource:0}: Error finding container 92220d90a71377bcd5d827a15b9fed7ba3d9763763331de829b01fa96e2a6b96: Status 404 returned error can't find the container with id 92220d90a71377bcd5d827a15b9fed7ba3d9763763331de829b01fa96e2a6b96 Oct 29 22:34:25.078210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.078189 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s"] Oct 29 22:34:25.137496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.137469 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjsdx\" (UniqueName: \"kubernetes.io/projected/b274dc39-92a7-4a00-894f-57b6314671c4-kube-api-access-cjsdx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:25.137609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.137499 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274dc39-92a7-4a00-894f-57b6314671c4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:25.706362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.706274 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" Oct 29 22:34:25.706544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.706277 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5" event={"ID":"b274dc39-92a7-4a00-894f-57b6314671c4","Type":"ContainerDied","Data":"0b635d0ffc39034572b2a70f1fb6983e9660f077435fbc136776f2085ccd7173"} Oct 29 22:34:25.706544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.706468 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b635d0ffc39034572b2a70f1fb6983e9660f077435fbc136776f2085ccd7173" Oct 29 22:34:25.708369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.708339 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" event={"ID":"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb","Type":"ContainerStarted","Data":"0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2"} Oct 29 22:34:25.708539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.708389 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" event={"ID":"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb","Type":"ContainerStarted","Data":"92220d90a71377bcd5d827a15b9fed7ba3d9763763331de829b01fa96e2a6b96"} Oct 29 22:34:25.708539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.708427 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:25.710670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.710649 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" event={"ID":"7c9579b0-9d8f-4caa-babc-69ba5a02a012","Type":"ContainerStarted","Data":"dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f"} Oct 29 22:34:25.710777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.710683 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" event={"ID":"7c9579b0-9d8f-4caa-babc-69ba5a02a012","Type":"ContainerStarted","Data":"dae6a914aadcb4def850d0fdcbf4b8660c2a4f34ba1ea40975dc8f01272bccc6"} Oct 29 22:34:25.710777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.710755 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:25.761342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.761301 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" podStartSLOduration=1.7612878730000001 podStartE2EDuration="1.761287873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:25.749108704 +0000 UTC m=+1132.643183625" watchObservedRunningTime="2026-10-29 22:34:25.761287873 +0000 UTC m=+1132.655362792" Oct 29 22:34:25.789057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:25.788986 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" podStartSLOduration=1.78897241 podStartE2EDuration="1.78897241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:25.788754537 +0000 UTC m=+1132.682829459" watchObservedRunningTime="2026-10-29 22:34:25.78897241 +0000 UTC m=+1132.683047329" Oct 29 22:34:27.542867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.542833 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz"] Oct 29 22:34:27.543564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.543549 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b274dc39-92a7-4a00-894f-57b6314671c4" containerName="revoke-keys" Oct 29 22:34:27.543610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.543565 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b274dc39-92a7-4a00-894f-57b6314671c4" containerName="revoke-keys" Oct 29 22:34:27.543722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.543712 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b274dc39-92a7-4a00-894f-57b6314671c4" containerName="revoke-keys" Oct 29 22:34:27.547772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.547755 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.569266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.569246 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-591986bb-openshift-default-dockercfg-4qtx4\"" Oct 29 22:34:27.588744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.588715 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz"] Oct 29 22:34:27.684593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.684556 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.684790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.684621 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2twj\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.684790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.684763 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.684890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.684818 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.684976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.684954 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.685010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.684986 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.685082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.685070 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.685185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.685174 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.685259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.685248 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794262 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794342 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794417 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2twj\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794494 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794548 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794634 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794663 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794735 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.794850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794793 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.795066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.794901 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.795183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.795113 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.796048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.796021 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.796140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.796086 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.796471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.796448 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.797797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.797778 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.798514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.798492 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.831748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.828020 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2twj\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.831748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.830263 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token\") pod \"e2e-derive-591986bb-openshift-default-cb894bd5-5thqz\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:27.862026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:27.861989 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:28.227014 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:28.226987 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e29c1c_7b1e_4adf_970f_9a876045babb.slice/crio-cfa51b3a5c99ecfb99317d3a00c7e0f2e0a44f569c0eddc9c52ea3d1d3a79b8f WatchSource:0}: Error finding container cfa51b3a5c99ecfb99317d3a00c7e0f2e0a44f569c0eddc9c52ea3d1d3a79b8f: Status 404 returned error can't find the container with id cfa51b3a5c99ecfb99317d3a00c7e0f2e0a44f569c0eddc9c52ea3d1d3a79b8f Oct 29 22:34:28.229273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.229250 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz"] Oct 29 22:34:28.229439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.229411 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:28.229487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.229473 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:28.229521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.229502 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:28.744766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.744727 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" event={"ID":"78e29c1c-7b1e-4adf-970f-9a876045babb","Type":"ContainerStarted","Data":"8271b9130bc6e0af0a6b9997a77591b955c69f4656964dd579a3ceb7e2a89787"} Oct 29 22:34:28.745227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.744780 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" event={"ID":"78e29c1c-7b1e-4adf-970f-9a876045babb","Type":"ContainerStarted","Data":"cfa51b3a5c99ecfb99317d3a00c7e0f2e0a44f569c0eddc9c52ea3d1d3a79b8f"} Oct 29 22:34:28.793750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.793687 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" podStartSLOduration=1.793675152 podStartE2EDuration="1.793675152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:28.788762434 +0000 UTC m=+1135.682837355" watchObservedRunningTime="2026-10-29 22:34:28.793675152 +0000 UTC m=+1135.687750073" Oct 29 22:34:28.867187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:28.867156 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:29.872968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:29.872939 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:30.791833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:30.791806 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:30.793953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:30.793931 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:34:32.539464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.539428 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r"] Oct 29 22:34:32.544452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.544430 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:32.575085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.575051 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-591986bb-dockercfg-96rfc\"" Oct 29 22:34:32.576867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.576839 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-591986bb\"" Oct 29 22:34:32.645794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.645746 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r"] Oct 29 22:34:32.875449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.756309 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf"] Oct 29 22:34:32.923764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.857981 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqnn9\" (UniqueName: \"kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9\") pod \"payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:32.948737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.935951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:32.994005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.993957 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf"] Oct 29 22:34:32.994478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:32.994459 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.227814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.224435 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:33.227814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.224552 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume\") pod \"payload-processing-e2e-derive-591986bb-8559c645b8-92szf\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.227814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.224672 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6\") pod \"payload-processing-e2e-derive-591986bb-8559c645b8-92szf\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.269510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.269467 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqnn9\" (UniqueName: \"kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9\") pod \"payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:33.330839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.330809 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-591986bb\"" Oct 29 22:34:33.380964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.380838 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqnn9\" (UniqueName: \"kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9\") pod \"payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:33.394603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.394560 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume\") pod \"payload-processing-e2e-derive-591986bb-8559c645b8-92szf\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.415995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.415953 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6\") pod \"payload-processing-e2e-derive-591986bb-8559c645b8-92szf\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.417038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.403251 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:33.417225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.395611 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume\") pod \"payload-processing-e2e-derive-591986bb-8559c645b8-92szf\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.492851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.477459 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6\") pod \"payload-processing-e2e-derive-591986bb-8559c645b8-92szf\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.560722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.558291 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-591986bb-dockercfg-96rfc\"" Oct 29 22:34:33.664158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.658684 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:33.669890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.669862 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:33.900789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.860602 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:33.900789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.877038 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:33.943767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.907959 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:33.966109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:33.966057 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.046828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.042658 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.113726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.060919 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.143058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.142224 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:34:34.205905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.205851 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.206221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.206192 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:34:34.480641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.478921 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.480641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.479269 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:34:34.570182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.569850 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:34.570182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.569909 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.771776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.771253 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:34:34.875327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.875285 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:34.876300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.876275 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:34:34.877464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.877437 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.879012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.878995 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:34:34.910852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.910810 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.925359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.925328 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:34:34.967777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.966799 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.967777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:34.967271 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:34:35.300196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.158483 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.300196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.158976 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:34:35.300196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.160416 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:34:35.300196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.181876 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:34:35.300196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.231113 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.328259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.300890 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.328259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.302677 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:34:35.328259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.302758 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:34:35.453229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.407643 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.453229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.407772 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 29 22:34:35.806739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.803051 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:35.807647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.807611 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:34:35.974932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.974822 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.996875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.979980 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.997165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:35.997128 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.064629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.064587 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" podUID="88e74199-7eba-4437-b7e7-d2dc4094d87e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 29 22:34:36.067724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.065159 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:34:36.236909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.198376 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:34:36.255508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.205785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.259769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.231559 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.275803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.260177 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": context deadline exceeded" Oct 29 22:34:36.313514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.289822 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.364731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.303265 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:34:36.386855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.313792 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.387755 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:34:36.371719 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.046s" Oct 29 22:34:36.388025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.388008 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:34:36.388233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.388221 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:34:36.388372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.388362 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:34:36.388503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.388493 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:34:36.388634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.388624 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:34:36.589288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.589238 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.880023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:36.879931 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:34:37.346849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.346798 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.506822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.026497 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:34:37.532799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.028609 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:37.646759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.028651 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.713800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.052093 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.714970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.111845 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.715901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.347098 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.716571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.367857 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:34:37.717173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.367912 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:34:37.743816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.432284 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:34:37.744615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.432142 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:34:37.773216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:37.026461 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:38.132780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.128074 5367 patch_prober.go:29] interesting pod/metrics-server-684967c95f-cf5zb container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.11:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:38.132780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.130245 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-684967c95f-cf5zb" podUID="29897fb3-27ec-4f39-8f74-f96c5cba04a1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.11:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.313792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.299512 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx292" podUID="88e74199-7eba-4437-b7e7-d2dc4094d87e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 29 22:34:38.586335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.518882 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" podUID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 29 22:34:38.646835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.590891 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.646835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.592124 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:34:38.728720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.712420 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.748722 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:34:38.740664 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.396s" Oct 29 22:34:38.749515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.749487 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:34:38.749657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.749582 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:34:38.749657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.749637 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:34:38.749804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.749711 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:34:38.749804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.749764 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:34:38.762715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.760353 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.762715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.760794 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:34:38.791719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:38.786850 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:39.071357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.071157 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:34:39.107159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.107126 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r"] Oct 29 22:34:39.107295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.107235 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:34:39.110840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.110820 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:34:39.111446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.111430 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:34:39.134719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.132661 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:34:39.134719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.132737 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:34:39.151019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.150934 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:34:39.244660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.244507 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf"] Oct 29 22:34:39.484913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.484875 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" event={"ID":"4f021dfe-de47-4e20-a024-30858ed4230b","Type":"ContainerStarted","Data":"bb31b67d41b0239e5c3e4ed59f4646f5be2d9f8bdcb91f9e9fce3c6fdd57a11f"} Oct 29 22:34:39.485108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.484968 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" event={"ID":"4f021dfe-de47-4e20-a024-30858ed4230b","Type":"ContainerStarted","Data":"f3c0bbdbb194140f6305972492334ed435b111506d35e0cd971cc71fb28c0000"} Oct 29 22:34:39.485108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.485001 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:39.487440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.487045 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" event={"ID":"e80875a1-4893-49d4-a5fe-32164c6e6efb","Type":"ContainerStarted","Data":"6386d82230efdc6ab21dc49f8228167f60b0349c80bbd04e9e18a3053b1b0048"} Oct 29 22:34:39.487440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.487108 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" event={"ID":"e80875a1-4893-49d4-a5fe-32164c6e6efb","Type":"ContainerStarted","Data":"916dd23f53c75fa32a9583350e1672b55486607dc980d98e9a94d2ee437e8a96"} Oct 29 22:34:39.487440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.487157 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:39.625223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.625168 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" podStartSLOduration=7.625154499 podStartE2EDuration="7.625154499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:39.619076899 +0000 UTC m=+1146.513151821" watchObservedRunningTime="2026-10-29 22:34:39.625154499 +0000 UTC m=+1146.519229418" Oct 29 22:34:39.664290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:39.664237 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" podStartSLOduration=7.664219814 podStartE2EDuration="7.664219814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:39.659204196 +0000 UTC m=+1146.553279119" watchObservedRunningTime="2026-10-29 22:34:39.664219814 +0000 UTC m=+1146.558294736" Oct 29 22:34:45.037918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.037882 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk"] Oct 29 22:34:45.050269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.050237 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374802a8-5kslk"] Oct 29 22:34:45.225989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.225957 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b304ba5-ce09-4a09-b445-ee9407e0cf33" path="/var/lib/kubelet/pods/3b304ba5-ce09-4a09-b445-ee9407e0cf33/volumes" Oct 29 22:34:45.688832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.688794 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6"] Oct 29 22:34:45.699246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.695504 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.725274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.725233 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:34:45.725452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.725352 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:34:45.725506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.725478 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:34:45.734044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.734007 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:34:45.735085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.735059 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6"] Oct 29 22:34:45.736380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.736351 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.736572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.736471 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf6g\" (UniqueName: \"kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.736572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.736566 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.838538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.838496 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.838855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.838830 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.838970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.838945 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf6g\" (UniqueName: \"kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.839386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.839348 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.841948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.841923 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:45.859068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:45.859044 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf6g\" (UniqueName: \"kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g\") pod \"maas-api-revoke-sub-6ece9a38-cnjz6\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:46.055788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:46.055760 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:46.361889 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:46.361857 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b258e4_447d_4e0c_a956_756391bb785d.slice/crio-d2c1dbf3fb6b93d608ded62a11e5c6d91ea9d7018be1a2a1b253bec9bfadd2f5 WatchSource:0}: Error finding container d2c1dbf3fb6b93d608ded62a11e5c6d91ea9d7018be1a2a1b253bec9bfadd2f5: Status 404 returned error can't find the container with id d2c1dbf3fb6b93d608ded62a11e5c6d91ea9d7018be1a2a1b253bec9bfadd2f5 Oct 29 22:34:46.363558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:46.363534 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6"] Oct 29 22:34:46.560381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:46.560350 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" event={"ID":"b0b258e4-447d-4e0c-a956-756391bb785d","Type":"ContainerStarted","Data":"d2c1dbf3fb6b93d608ded62a11e5c6d91ea9d7018be1a2a1b253bec9bfadd2f5"} Oct 29 22:34:46.722334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:46.722307 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:34:47.034436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.034340 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6"] Oct 29 22:34:47.041350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.041006 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d318ddbd-b7xb6"] Oct 29 22:34:47.046669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.046639 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr"] Oct 29 22:34:47.051872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.051845 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388199dc-7fjmr"] Oct 29 22:34:47.224499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.224463 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838a0363-0d5a-489a-b097-10b15467be93" path="/var/lib/kubelet/pods/838a0363-0d5a-489a-b097-10b15467be93/volumes" Oct 29 22:34:47.225152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.225133 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caad35a0-c508-401b-b5ad-cc4374f10218" path="/var/lib/kubelet/pods/caad35a0-c508-401b-b5ad-cc4374f10218/volumes" Oct 29 22:34:47.572233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.572203 5367 generic.go:356] "Generic (PLEG): container finished" podID="b0b258e4-447d-4e0c-a956-756391bb785d" containerID="8c0d1678c92e54226090e7cf1f6720c658b71b5444047980a5a8d4059edc230e" exitCode=0 Oct 29 22:34:47.572398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:47.572292 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" event={"ID":"b0b258e4-447d-4e0c-a956-756391bb785d","Type":"ContainerDied","Data":"8c0d1678c92e54226090e7cf1f6720c658b71b5444047980a5a8d4059edc230e"} Oct 29 22:34:48.762622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.762594 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:48.889585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.889486 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca\") pod \"b0b258e4-447d-4e0c-a956-756391bb785d\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " Oct 29 22:34:48.889770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.889750 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g\" (UniqueName: \"kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g\") pod \"b0b258e4-447d-4e0c-a956-756391bb785d\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " Oct 29 22:34:48.889889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.889874 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca\") pod \"b0b258e4-447d-4e0c-a956-756391bb785d\" (UID: \"b0b258e4-447d-4e0c-a956-756391bb785d\") " Oct 29 22:34:48.890254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.890226 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca" pod "b0b258e4-447d-4e0c-a956-756391bb785d" (UID: "b0b258e4-447d-4e0c-a956-756391bb785d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:48.890682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.890661 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:48.893452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.893424 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca" pod "b0b258e4-447d-4e0c-a956-756391bb785d" (UID: "b0b258e4-447d-4e0c-a956-756391bb785d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:48.893778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.893753 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g" pod "b0b258e4-447d-4e0c-a956-756391bb785d" (UID: "b0b258e4-447d-4e0c-a956-756391bb785d"). InnerVolumeSpecName "kube-api-access-5pf6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:48.992975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.992938 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b258e4-447d-4e0c-a956-756391bb785d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:48.992975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:48.992976 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pf6g\" (UniqueName: \"kubernetes.io/projected/b0b258e4-447d-4e0c-a956-756391bb785d-kube-api-access-5pf6g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:49.593922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:49.593894 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" Oct 29 22:34:49.594118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:49.593934 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6" event={"ID":"b0b258e4-447d-4e0c-a956-756391bb785d","Type":"ContainerDied","Data":"d2c1dbf3fb6b93d608ded62a11e5c6d91ea9d7018be1a2a1b253bec9bfadd2f5"} Oct 29 22:34:49.594118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:49.593989 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c1dbf3fb6b93d608ded62a11e5c6d91ea9d7018be1a2a1b253bec9bfadd2f5" Oct 29 22:34:50.499581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:50.499552 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:34:50.500151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:50.500008 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:34:52.333171 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:34:52.333131 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:34:55.032193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:55.032153 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9"] Oct 29 22:34:55.044592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:55.044558 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c23eab8-ncjs9"] Oct 29 22:34:55.224009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:55.223976 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe9e46c-3a00-424a-bc25-55e500ff72b6" path="/var/lib/kubelet/pods/7fe9e46c-3a00-424a-bc25-55e500ff72b6/volumes" Oct 29 22:34:57.009843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.009812 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc"] Oct 29 22:34:57.010743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.010720 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b0b258e4-447d-4e0c-a956-756391bb785d" containerName="revoke-keys" Oct 29 22:34:57.010743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.010742 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b258e4-447d-4e0c-a956-756391bb785d" containerName="revoke-keys" Oct 29 22:34:57.010991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.010977 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b258e4-447d-4e0c-a956-756391bb785d" containerName="revoke-keys" Oct 29 22:34:57.015389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.015369 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.032435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.032405 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht"] Oct 29 22:34:57.033301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.033278 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlqm\" (UniqueName: \"kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.033360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.033344 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.033673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.033651 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.048672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.048641 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:34:57.049435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.049390 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:34:57.056994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.056963 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc"] Oct 29 22:34:57.063384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.063357 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99df198a-7d6ht"] Oct 29 22:34:57.107431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.107402 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:34:57.107431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.107417 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:34:57.134820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.134785 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.134980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.134897 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlqm\" (UniqueName: \"kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.134980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.134957 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.135645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.135621 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.137347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.137326 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.155357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.155328 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlqm\" (UniqueName: \"kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm\") pod \"maas-api-revoke-keys-e2e-ait-1309d608-hd5gc\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.224508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.224475 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8744b9f9-35c7-4a42-a6bd-adebcd1790c3" path="/var/lib/kubelet/pods/8744b9f9-35c7-4a42-a6bd-adebcd1790c3/volumes" Oct 29 22:34:57.357379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.357340 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:57.669042 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:34:57.668993 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aedbeb_05cd_4c0a_8a9a_7de87bf63582.slice/crio-6496f028bfd55f8a22092a6afd015cf009835450bb2bc4413632999c918ddbc9 WatchSource:0}: Error finding container 6496f028bfd55f8a22092a6afd015cf009835450bb2bc4413632999c918ddbc9: Status 404 returned error can't find the container with id 6496f028bfd55f8a22092a6afd015cf009835450bb2bc4413632999c918ddbc9 Oct 29 22:34:57.671264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:57.671239 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc"] Oct 29 22:34:58.697190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:58.697152 5367 generic.go:356] "Generic (PLEG): container finished" podID="44aedbeb-05cd-4c0a-8a9a-7de87bf63582" containerID="ec9e2e1d09eb95bee9634abc7d3fd6b3dd7416a1a9510132d0fee3fee149f339" exitCode=0 Oct 29 22:34:58.697584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:58.697237 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" event={"ID":"44aedbeb-05cd-4c0a-8a9a-7de87bf63582","Type":"ContainerDied","Data":"ec9e2e1d09eb95bee9634abc7d3fd6b3dd7416a1a9510132d0fee3fee149f339"} Oct 29 22:34:58.697584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:58.697313 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" event={"ID":"44aedbeb-05cd-4c0a-8a9a-7de87bf63582","Type":"ContainerStarted","Data":"6496f028bfd55f8a22092a6afd015cf009835450bb2bc4413632999c918ddbc9"} Oct 29 22:34:59.912773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.912746 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:34:59.981607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.981514 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca\") pod \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " Oct 29 22:34:59.981854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.981831 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca\") pod \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " Oct 29 22:34:59.981940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.981875 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm\" (UniqueName: \"kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm\") pod \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\" (UID: \"44aedbeb-05cd-4c0a-8a9a-7de87bf63582\") " Oct 29 22:34:59.982250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.982222 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca" pod "44aedbeb-05cd-4c0a-8a9a-7de87bf63582" (UID: "44aedbeb-05cd-4c0a-8a9a-7de87bf63582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:59.982858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.982773 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:34:59.984570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.984536 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca" pod "44aedbeb-05cd-4c0a-8a9a-7de87bf63582" (UID: "44aedbeb-05cd-4c0a-8a9a-7de87bf63582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:59.984804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:34:59.984779 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm" pod "44aedbeb-05cd-4c0a-8a9a-7de87bf63582" (UID: "44aedbeb-05cd-4c0a-8a9a-7de87bf63582"). InnerVolumeSpecName "kube-api-access-gnlqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:00.084050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:00.084007 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:00.084050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:00.084040 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnlqm\" (UniqueName: \"kubernetes.io/projected/44aedbeb-05cd-4c0a-8a9a-7de87bf63582-kube-api-access-gnlqm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:00.722107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:00.722077 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" Oct 29 22:35:00.722289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:00.722085 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc" event={"ID":"44aedbeb-05cd-4c0a-8a9a-7de87bf63582","Type":"ContainerDied","Data":"6496f028bfd55f8a22092a6afd015cf009835450bb2bc4413632999c918ddbc9"} Oct 29 22:35:00.722289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:00.722203 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6496f028bfd55f8a22092a6afd015cf009835450bb2bc4413632999c918ddbc9" Oct 29 22:35:01.044418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:01.044337 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9"] Oct 29 22:35:01.050852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:01.050819 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b6331c-9wlb9"] Oct 29 22:35:01.223736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:01.223661 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296071a7-c807-410f-ad09-8a97c61ffb09" path="/var/lib/kubelet/pods/296071a7-c807-410f-ad09-8a97c61ffb09/volumes" Oct 29 22:35:05.035783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:05.035751 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c"] Oct 29 22:35:05.046412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:05.046383 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41a370b-vqp6c"] Oct 29 22:35:05.245951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:05.245910 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e979399-1cdc-404f-93c6-edfb5958da08" path="/var/lib/kubelet/pods/0e979399-1cdc-404f-93c6-edfb5958da08/volumes" Oct 29 22:35:08.798791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.798756 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx"] Oct 29 22:35:08.799919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.799892 5367 cpu_manager.go:414] "containerMap: removing container" podUID="44aedbeb-05cd-4c0a-8a9a-7de87bf63582" containerName="revoke-keys" Oct 29 22:35:08.799919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.799912 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44aedbeb-05cd-4c0a-8a9a-7de87bf63582" containerName="revoke-keys" Oct 29 22:35:08.800113 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.800029 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="44aedbeb-05cd-4c0a-8a9a-7de87bf63582" containerName="revoke-keys" Oct 29 22:35:08.805912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.805895 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:08.821326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.821300 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx"] Oct 29 22:35:08.827159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.827129 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:35:08.827277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.827183 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:35:08.827352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.827285 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:35:08.837017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.836991 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:35:08.933483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.933443 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:08.933879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.933857 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:08.934073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:08.934029 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.037275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.037230 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.037487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.037299 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.037592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.037575 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.039063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.039031 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.057158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.057110 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.065184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.065155 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5\") pod \"maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.126433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.126384 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:09.479542 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:35:09.479506 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f177190_b19c_48a7_93bb_bdc9ab2cd117.slice/crio-9bec04ad69ed52325d85ffa40dd913501efa25104b48090fa481abc48436b370 WatchSource:0}: Error finding container 9bec04ad69ed52325d85ffa40dd913501efa25104b48090fa481abc48436b370: Status 404 returned error can't find the container with id 9bec04ad69ed52325d85ffa40dd913501efa25104b48090fa481abc48436b370 Oct 29 22:35:09.480961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.480934 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx"] Oct 29 22:35:09.814788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:09.814751 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" event={"ID":"1f177190-b19c-48a7-93bb-bdc9ab2cd117","Type":"ContainerStarted","Data":"9bec04ad69ed52325d85ffa40dd913501efa25104b48090fa481abc48436b370"} Oct 29 22:35:10.046018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:10.045979 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg"] Oct 29 22:35:10.058463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:10.058400 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca3013a-w48fg"] Oct 29 22:35:10.878731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:10.877205 5367 generic.go:356] "Generic (PLEG): container finished" podID="1f177190-b19c-48a7-93bb-bdc9ab2cd117" containerID="7be1d2c3a10c1cf170627379c4041cf6f9ffa192f0cc4c0900bb8e14fb31bbc4" exitCode=0 Oct 29 22:35:10.878731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:10.877415 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" event={"ID":"1f177190-b19c-48a7-93bb-bdc9ab2cd117","Type":"ContainerDied","Data":"7be1d2c3a10c1cf170627379c4041cf6f9ffa192f0cc4c0900bb8e14fb31bbc4"} Oct 29 22:35:11.087863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:11.087821 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc"] Oct 29 22:35:11.096358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:11.096321 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2e67c2-c4pgc"] Oct 29 22:35:11.228222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:11.228139 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e166181-5ba6-442f-aa45-8a4b98c13f0d" path="/var/lib/kubelet/pods/0e166181-5ba6-442f-aa45-8a4b98c13f0d/volumes" Oct 29 22:35:11.228765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:11.228738 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc3f248-554f-4b40-ac9f-9f36c2c049e9" path="/var/lib/kubelet/pods/abc3f248-554f-4b40-ac9f-9f36c2c049e9/volumes" Oct 29 22:35:12.729896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:12.728800 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.777604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:12.776794 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.888722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:12.862791 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.089721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.088975 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.227723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.214850 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:35:13.227723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.225030 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Oct 29 22:35:13.227723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.227122 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.301420 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.301374 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.435774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.328198 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:35:13.435774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.434863 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.435774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.328445 5367 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:35:13.435774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.434985 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.653737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.640425 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.653737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.640808 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.713337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.713226 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.714094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.714057 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:35:13.732844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.732802 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.923783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.922120 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.933957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:13.933918 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:35:14.015717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.015526 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:14.039539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.038183 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca\") pod \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " Oct 29 22:35:14.039539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.038264 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca\") pod \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " Oct 29 22:35:14.039539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.038464 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5\") pod \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\" (UID: \"1f177190-b19c-48a7-93bb-bdc9ab2cd117\") " Oct 29 22:35:14.040435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.040412 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca" pod "1f177190-b19c-48a7-93bb-bdc9ab2cd117" (UID: "1f177190-b19c-48a7-93bb-bdc9ab2cd117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:14.067497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.067455 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" event={"ID":"1f177190-b19c-48a7-93bb-bdc9ab2cd117","Type":"ContainerDied","Data":"9bec04ad69ed52325d85ffa40dd913501efa25104b48090fa481abc48436b370"} Oct 29 22:35:14.067633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.067529 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bec04ad69ed52325d85ffa40dd913501efa25104b48090fa481abc48436b370" Oct 29 22:35:14.067633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.067588 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:35:14.067633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.067626 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:35:14.067808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.067663 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:35:14.089203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.089152 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:14.089569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.089529 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:35:14.089677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.089600 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:35:14.228895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.223611 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:14.247460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.247405 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca" pod "1f177190-b19c-48a7-93bb-bdc9ab2cd117" (UID: "1f177190-b19c-48a7-93bb-bdc9ab2cd117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:14.284321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.284244 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:35:14.322738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.313033 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:35:14.369658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.369544 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f177190-b19c-48a7-93bb-bdc9ab2cd117-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:14.371068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.371043 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5" pod "1f177190-b19c-48a7-93bb-bdc9ab2cd117" (UID: "1f177190-b19c-48a7-93bb-bdc9ab2cd117"). InnerVolumeSpecName "kube-api-access-4pzr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:14.380963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.378850 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:35:14.482730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.482043 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/1f177190-b19c-48a7-93bb-bdc9ab2cd117-kube-api-access-4pzr5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:14.939761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:14.939685 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx" Oct 29 22:35:19.873589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.873558 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9"] Oct 29 22:35:19.874293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.874277 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1f177190-b19c-48a7-93bb-bdc9ab2cd117" containerName="revoke-keys" Oct 29 22:35:19.874293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.874293 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f177190-b19c-48a7-93bb-bdc9ab2cd117" containerName="revoke-keys" Oct 29 22:35:19.874466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.874455 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f177190-b19c-48a7-93bb-bdc9ab2cd117" containerName="revoke-keys" Oct 29 22:35:19.878692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.878667 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.904637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.904605 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9"] Oct 29 22:35:19.909437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.909410 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3593b8ba-gw-openshift-default-dockercfg-skwqf\"" Oct 29 22:35:19.923737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.923648 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.923922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.923854 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.923922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.923903 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.924035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.923938 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.924035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.923984 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.924148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.924126 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvrn\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.924225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.924210 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.924294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.924282 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:19.924402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:19.924387 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.025621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025587 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.025853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025649 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.025853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025691 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.025853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025738 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025900 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025946 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.025981 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.026030 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.026153 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.026165 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvrn\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.026254 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.026343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.026783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.026659 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.028552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.028528 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.031256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.031232 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.031921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.031899 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.033492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.033468 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh"] Oct 29 22:35:20.041556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.041530 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d263071-5jbgh"] Oct 29 22:35:20.046151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.046126 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.046372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.046342 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvrn\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.193574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.193495 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:20.551619 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:35:20.551590 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00116d89_c80a_409d_b395_27b4a2bb2b98.slice/crio-b931f32c29b2687e7c7a284573180f534c35713b6a106eb0938163b7f7492b09 WatchSource:0}: Error finding container b931f32c29b2687e7c7a284573180f534c35713b6a106eb0938163b7f7492b09: Status 404 returned error can't find the container with id b931f32c29b2687e7c7a284573180f534c35713b6a106eb0938163b7f7492b09 Oct 29 22:35:20.554308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.554272 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:35:20.554438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.554339 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:35:20.554438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.554369 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:35:20.556304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:20.556282 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9"] Oct 29 22:35:21.003764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:21.003721 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" event={"ID":"00116d89-c80a-409d-b395-27b4a2bb2b98","Type":"ContainerStarted","Data":"b6739c59cf5a9aaedbb4246fa057c6b10358cac1e089a47a77b17453ffbdc77c"} Oct 29 22:35:21.004261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:21.003818 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" event={"ID":"00116d89-c80a-409d-b395-27b4a2bb2b98","Type":"ContainerStarted","Data":"b931f32c29b2687e7c7a284573180f534c35713b6a106eb0938163b7f7492b09"} Oct 29 22:35:21.194431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:21.194395 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:21.195658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:21.195618 5367 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:35:21.195826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:21.195745 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 29 22:35:21.229989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:21.229954 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c16e75e-807e-41b8-9911-5e007362a202" path="/var/lib/kubelet/pods/0c16e75e-807e-41b8-9911-5e007362a202/volumes" Oct 29 22:35:22.181725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:22.178613 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podStartSLOduration=3.178591468 podStartE2EDuration="3.178591468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:21.073781676 +0000 UTC m=+1187.967856597" watchObservedRunningTime="2026-10-29 22:35:22.178591468 +0000 UTC m=+1189.072666389" Oct 29 22:35:22.190881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:22.190847 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc"] Oct 29 22:35:22.194142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:22.194109 5367 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:35:22.194293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:22.194182 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 29 22:35:22.200276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:22.200213 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1309d608-hd5gc"] Oct 29 22:35:22.410738 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:35:22.405713 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:35:23.036112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:23.036076 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9"] Oct 29 22:35:23.044932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:23.044896 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3f44c-d5xp9"] Oct 29 22:35:23.195156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:23.194842 5367 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:35:23.195156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:23.194901 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 29 22:35:23.227904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:23.227867 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44aedbeb-05cd-4c0a-8a9a-7de87bf63582" path="/var/lib/kubelet/pods/44aedbeb-05cd-4c0a-8a9a-7de87bf63582/volumes" Oct 29 22:35:23.228732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:23.228480 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48c0557-8c59-4020-881c-51feda119490" path="/var/lib/kubelet/pods/d48c0557-8c59-4020-881c-51feda119490/volumes" Oct 29 22:35:24.194939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:24.194903 5367 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:35:24.195124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:24.194994 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 29 22:35:25.193817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:25.193784 5367 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:35:25.194211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:25.193841 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 29 22:35:26.194796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:26.194757 5367 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:35:26.195327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:26.194829 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 29 22:35:27.201612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:27.201456 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:27.203170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:27.203012 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:27.203735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:27.203678 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:29.405152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.405123 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b"] Oct 29 22:35:29.411143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.411122 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:29.429740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.429714 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b"] Oct 29 22:35:29.435937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.435905 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e1d9591d\"" Oct 29 22:35:29.501473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.501443 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:29.501718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.501652 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7glj\" (UniqueName: \"kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj\") pod \"payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:29.696263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.695600 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7glj\" (UniqueName: \"kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj\") pod \"payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:29.696263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.695968 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:29.696690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.696667 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:29.744325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:29.744283 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7glj\" (UniqueName: \"kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj\") pod \"payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:30.027974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:30.027893 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:30.355008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:30.354908 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b"] Oct 29 22:35:31.041762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.041727 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n"] Oct 29 22:35:31.054554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.054524 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23d743-6kn4n"] Oct 29 22:35:31.225979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.225945 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ce4bae-2082-46cb-a5ee-b1350c010d17" path="/var/lib/kubelet/pods/d4ce4bae-2082-46cb-a5ee-b1350c010d17/volumes" Oct 29 22:35:31.230015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.229984 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" event={"ID":"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8","Type":"ContainerStarted","Data":"8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1"} Oct 29 22:35:31.230186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.230042 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" event={"ID":"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8","Type":"ContainerStarted","Data":"bde4296596d76ff0856410990d7ffc6b76ceb9c07b54fe63d493ac011b718cb7"} Oct 29 22:35:31.230186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.230087 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:31.276738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:31.276641 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" podStartSLOduration=2.276625875 podStartE2EDuration="2.276625875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:31.269272126 +0000 UTC m=+1198.163347048" watchObservedRunningTime="2026-10-29 22:35:31.276625875 +0000 UTC m=+1198.170700794" Oct 29 22:35:34.990935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:34.990902 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx"] Oct 29 22:35:35.003006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:35.002977 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3593b8ba-j4fwx"] Oct 29 22:35:35.141425 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:35.141396 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9"] Oct 29 22:35:35.141632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:35.141598 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" containerID="cri-o://b6739c59cf5a9aaedbb4246fa057c6b10358cac1e089a47a77b17453ffbdc77c" gracePeriod=30 Oct 29 22:35:35.237747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:35.233983 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f177190-b19c-48a7-93bb-bdc9ab2cd117" path="/var/lib/kubelet/pods/1f177190-b19c-48a7-93bb-bdc9ab2cd117/volumes" Oct 29 22:35:36.685018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.684983 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5"] Oct 29 22:35:36.691206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.691184 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.707555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.707527 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:35:36.709869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.709843 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5"] Oct 29 22:35:36.715949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.715877 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:35:36.716074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.715975 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:35:36.716280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.716261 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:35:36.830393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.830362 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6lj\" (UniqueName: \"kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.830577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.830493 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.830772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.830754 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.932524 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.932490 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.932727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.932598 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6lj\" (UniqueName: \"kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.932727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.932668 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.933230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.933211 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.935069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.935014 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:36.952836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:36.952812 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6lj\" (UniqueName: \"kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj\") pod \"maas-api-revoke-keys-e2e-derive-591986bb-z4lf5\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:37.007207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:37.007182 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:37.316033 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:35:37.315991 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408d80f0_2c15_4dbe_ad81_1f4f852c8004.slice/crio-91c9f8f18a1ed3fe555244b1c259b8074e860c9c2a851ee2b68ff25b2af6024e WatchSource:0}: Error finding container 91c9f8f18a1ed3fe555244b1c259b8074e860c9c2a851ee2b68ff25b2af6024e: Status 404 returned error can't find the container with id 91c9f8f18a1ed3fe555244b1c259b8074e860c9c2a851ee2b68ff25b2af6024e Oct 29 22:35:37.318875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:37.318850 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5"] Oct 29 22:35:38.309350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:38.309304 5367 generic.go:356] "Generic (PLEG): container finished" podID="408d80f0-2c15-4dbe-ad81-1f4f852c8004" containerID="b5f682878d526cd0a92551a9861295be20fac6306bd6beda3ae88f5bbc554172" exitCode=0 Oct 29 22:35:38.309350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:38.309346 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" event={"ID":"408d80f0-2c15-4dbe-ad81-1f4f852c8004","Type":"ContainerDied","Data":"b5f682878d526cd0a92551a9861295be20fac6306bd6beda3ae88f5bbc554172"} Oct 29 22:35:38.309787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:38.309397 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" event={"ID":"408d80f0-2c15-4dbe-ad81-1f4f852c8004","Type":"ContainerStarted","Data":"91c9f8f18a1ed3fe555244b1c259b8074e860c9c2a851ee2b68ff25b2af6024e"} Oct 29 22:35:39.497524 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.497482 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:39.681226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.681140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca\") pod \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " Oct 29 22:35:39.681397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.681286 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj\" (UniqueName: \"kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj\") pod \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " Oct 29 22:35:39.681397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.681342 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca\") pod \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\" (UID: \"408d80f0-2c15-4dbe-ad81-1f4f852c8004\") " Oct 29 22:35:39.681575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.681550 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca" pod "408d80f0-2c15-4dbe-ad81-1f4f852c8004" (UID: "408d80f0-2c15-4dbe-ad81-1f4f852c8004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:39.681888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.681868 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:39.683405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.683386 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca" pod "408d80f0-2c15-4dbe-ad81-1f4f852c8004" (UID: "408d80f0-2c15-4dbe-ad81-1f4f852c8004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:39.683466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.683449 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj" pod "408d80f0-2c15-4dbe-ad81-1f4f852c8004" (UID: "408d80f0-2c15-4dbe-ad81-1f4f852c8004"). InnerVolumeSpecName "kube-api-access-lh6lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:39.783048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.783020 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408d80f0-2c15-4dbe-ad81-1f4f852c8004-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:39.783048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:39.783045 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6lj\" (UniqueName: \"kubernetes.io/projected/408d80f0-2c15-4dbe-ad81-1f4f852c8004-kube-api-access-lh6lj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.339089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.339051 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" event={"ID":"408d80f0-2c15-4dbe-ad81-1f4f852c8004","Type":"ContainerDied","Data":"91c9f8f18a1ed3fe555244b1c259b8074e860c9c2a851ee2b68ff25b2af6024e"} Oct 29 22:35:40.339332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.339108 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c9f8f18a1ed3fe555244b1c259b8074e860c9c2a851ee2b68ff25b2af6024e" Oct 29 22:35:40.339332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.339169 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5" Oct 29 22:35:40.343911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.343881 5367 generic.go:356] "Generic (PLEG): container finished" podID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerID="b6739c59cf5a9aaedbb4246fa057c6b10358cac1e089a47a77b17453ffbdc77c" exitCode=0 Oct 29 22:35:40.344269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.344247 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" event={"ID":"00116d89-c80a-409d-b395-27b4a2bb2b98","Type":"ContainerDied","Data":"b6739c59cf5a9aaedbb4246fa057c6b10358cac1e089a47a77b17453ffbdc77c"} Oct 29 22:35:40.442523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.442492 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:40.597101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.597070 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.598832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.597163 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.598832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.597745 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:40.598832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.597867 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.598832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.598136 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.598832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.598197 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.599616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.599588 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.599815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.599653 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.599908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.599874 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.599908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.599890 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:40.600089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.599969 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.600089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.599982 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.600089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600000 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data\") pod \"00116d89-c80a-409d-b395-27b4a2bb2b98\" (UID: \"00116d89-c80a-409d-b395-27b4a2bb2b98\") " Oct 29 22:35:40.600241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600018 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.600528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600400 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.600853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600834 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.600929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600856 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.600929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600869 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00116d89-c80a-409d-b395-27b4a2bb2b98-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.600929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600882 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.600929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600894 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.600929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.600907 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.604222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.604192 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "kube-api-access-gpvrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:40.604454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.604431 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.604454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.604431 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo" pod "00116d89-c80a-409d-b395-27b4a2bb2b98" (UID: "00116d89-c80a-409d-b395-27b4a2bb2b98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:35:40.702493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.702403 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.702493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.702435 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvrn\" (UniqueName: \"kubernetes.io/projected/00116d89-c80a-409d-b395-27b4a2bb2b98-kube-api-access-gpvrn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.702493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:40.702463 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00116d89-c80a-409d-b395-27b4a2bb2b98-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:41.355143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:41.355112 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" Oct 29 22:35:41.355331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:41.355149 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9" event={"ID":"00116d89-c80a-409d-b395-27b4a2bb2b98","Type":"ContainerDied","Data":"b931f32c29b2687e7c7a284573180f534c35713b6a106eb0938163b7f7492b09"} Oct 29 22:35:41.355331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:41.355216 5367 scope.go:122] "RemoveContainer" containerID="b6739c59cf5a9aaedbb4246fa057c6b10358cac1e089a47a77b17453ffbdc77c" Oct 29 22:35:41.427227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:41.427200 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9"] Oct 29 22:35:41.439791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:41.439767 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-msvx9"] Oct 29 22:35:42.245106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:42.245079 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:35:43.224508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:43.224477 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" path="/var/lib/kubelet/pods/00116d89-c80a-409d-b395-27b4a2bb2b98/volumes" Oct 29 22:35:47.074329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:47.074292 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf"] Oct 29 22:35:47.109678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:47.109645 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r"] Oct 29 22:35:51.684194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:51.684158 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5"] Oct 29 22:35:51.692132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:51.692106 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-591986bb-z4lf5"] Oct 29 22:35:51.898159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:51.898111 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz"] Oct 29 22:35:51.899168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:51.899135 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" podUID="78e29c1c-7b1e-4adf-970f-9a876045babb" containerName="istio-proxy" containerID="cri-o://8271b9130bc6e0af0a6b9997a77591b955c69f4656964dd579a3ceb7e2a89787" gracePeriod=30 Oct 29 22:35:52.081721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.077772 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" podUID="e80875a1-4893-49d4-a5fe-32164c6e6efb" containerName="payload-processing" containerID="cri-o://6386d82230efdc6ab21dc49f8228167f60b0349c80bbd04e9e18a3053b1b0048" gracePeriod=25 Oct 29 22:35:52.113948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.113909 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" podUID="4f021dfe-de47-4e20-a024-30858ed4230b" containerName="payload-pre-processing" containerID="cri-o://bb31b67d41b0239e5c3e4ed59f4646f5be2d9f8bdcb91f9e9fce3c6fdd57a11f" gracePeriod=25 Oct 29 22:35:52.491723 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:35:52.491439 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:35:52.521558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.519320 5367 generic.go:356] "Generic (PLEG): container finished" podID="4f021dfe-de47-4e20-a024-30858ed4230b" containerID="bb31b67d41b0239e5c3e4ed59f4646f5be2d9f8bdcb91f9e9fce3c6fdd57a11f" exitCode=0 Oct 29 22:35:52.521558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.519439 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" event={"ID":"4f021dfe-de47-4e20-a024-30858ed4230b","Type":"ContainerDied","Data":"bb31b67d41b0239e5c3e4ed59f4646f5be2d9f8bdcb91f9e9fce3c6fdd57a11f"} Oct 29 22:35:52.521558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.521453 5367 generic.go:356] "Generic (PLEG): container finished" podID="e80875a1-4893-49d4-a5fe-32164c6e6efb" containerID="6386d82230efdc6ab21dc49f8228167f60b0349c80bbd04e9e18a3053b1b0048" exitCode=0 Oct 29 22:35:52.521828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.521587 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" event={"ID":"e80875a1-4893-49d4-a5fe-32164c6e6efb","Type":"ContainerDied","Data":"6386d82230efdc6ab21dc49f8228167f60b0349c80bbd04e9e18a3053b1b0048"} Oct 29 22:35:52.684532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.684500 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:35:52.686598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.686579 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:35:52.822948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.822909 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6\") pod \"e80875a1-4893-49d4-a5fe-32164c6e6efb\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " Oct 29 22:35:52.823111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.823072 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9\" (UniqueName: \"kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9\") pod \"4f021dfe-de47-4e20-a024-30858ed4230b\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " Oct 29 22:35:52.823242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.823225 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume\") pod \"e80875a1-4893-49d4-a5fe-32164c6e6efb\" (UID: \"e80875a1-4893-49d4-a5fe-32164c6e6efb\") " Oct 29 22:35:52.823304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.823280 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume\") pod \"4f021dfe-de47-4e20-a024-30858ed4230b\" (UID: \"4f021dfe-de47-4e20-a024-30858ed4230b\") " Oct 29 22:35:52.823770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.823740 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume" pod "e80875a1-4893-49d4-a5fe-32164c6e6efb" (UID: "e80875a1-4893-49d4-a5fe-32164c6e6efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:52.823934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.823912 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume" pod "4f021dfe-de47-4e20-a024-30858ed4230b" (UID: "4f021dfe-de47-4e20-a024-30858ed4230b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:52.824122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.824102 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80875a1-4893-49d4-a5fe-32164c6e6efb-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:52.824201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.824127 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021dfe-de47-4e20-a024-30858ed4230b-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:52.825848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.825820 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6" pod "e80875a1-4893-49d4-a5fe-32164c6e6efb" (UID: "e80875a1-4893-49d4-a5fe-32164c6e6efb"). InnerVolumeSpecName "kube-api-access-wkzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:52.826060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.826040 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9" pod "4f021dfe-de47-4e20-a024-30858ed4230b" (UID: "4f021dfe-de47-4e20-a024-30858ed4230b"). InnerVolumeSpecName "kube-api-access-pqnn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:52.926033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.925957 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/e80875a1-4893-49d4-a5fe-32164c6e6efb-kube-api-access-wkzz6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:52.926033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:52.925983 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqnn9\" (UniqueName: \"kubernetes.io/projected/4f021dfe-de47-4e20-a024-30858ed4230b-kube-api-access-pqnn9\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:35:53.033792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.033756 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw"] Oct 29 22:35:53.049301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.049250 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9"] Oct 29 22:35:53.079317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.079280 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c530e-b9mmw"] Oct 29 22:35:53.084935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.084906 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5277424d-q5tk9"] Oct 29 22:35:53.225864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.225789 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129e0971-b687-4d26-ac9e-5615d099cc2e" path="/var/lib/kubelet/pods/129e0971-b687-4d26-ac9e-5615d099cc2e/volumes" Oct 29 22:35:53.226247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.226231 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fd40b7-c6d7-408b-8718-cdf84853cdd9" path="/var/lib/kubelet/pods/14fd40b7-c6d7-408b-8718-cdf84853cdd9/volumes" Oct 29 22:35:53.226686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.226672 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408d80f0-2c15-4dbe-ad81-1f4f852c8004" path="/var/lib/kubelet/pods/408d80f0-2c15-4dbe-ad81-1f4f852c8004/volumes" Oct 29 22:35:53.534293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.534198 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" Oct 29 22:35:53.534293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.534213 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf" event={"ID":"e80875a1-4893-49d4-a5fe-32164c6e6efb","Type":"ContainerDied","Data":"916dd23f53c75fa32a9583350e1672b55486607dc980d98e9a94d2ee437e8a96"} Oct 29 22:35:53.534293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.534277 5367 scope.go:122] "RemoveContainer" containerID="6386d82230efdc6ab21dc49f8228167f60b0349c80bbd04e9e18a3053b1b0048" Oct 29 22:35:53.536597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.536574 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" Oct 29 22:35:53.536850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.536574 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r" event={"ID":"4f021dfe-de47-4e20-a024-30858ed4230b","Type":"ContainerDied","Data":"f3c0bbdbb194140f6305972492334ed435b111506d35e0cd971cc71fb28c0000"} Oct 29 22:35:53.546541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.546523 5367 scope.go:122] "RemoveContainer" containerID="bb31b67d41b0239e5c3e4ed59f4646f5be2d9f8bdcb91f9e9fce3c6fdd57a11f" Oct 29 22:35:53.602203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.602178 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r"] Oct 29 22:35:53.615127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.615102 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-591986bb-b7fb6c67f-tx58r"] Oct 29 22:35:53.694801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.693345 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf"] Oct 29 22:35:53.705764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:53.705731 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-591986bb-8559c645b8-92szf"] Oct 29 22:35:54.030829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:54.030798 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g"] Oct 29 22:35:54.036336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:54.036311 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee47d56-kl98g"] Oct 29 22:35:55.226115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:55.226083 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f021dfe-de47-4e20-a024-30858ed4230b" path="/var/lib/kubelet/pods/4f021dfe-de47-4e20-a024-30858ed4230b/volumes" Oct 29 22:35:55.226807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:55.226787 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80875a1-4893-49d4-a5fe-32164c6e6efb" path="/var/lib/kubelet/pods/e80875a1-4893-49d4-a5fe-32164c6e6efb/volumes" Oct 29 22:35:55.227238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:55.227223 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8a9438-9aba-4ea9-badd-f31bf8b77115" path="/var/lib/kubelet/pods/fe8a9438-9aba-4ea9-badd-f31bf8b77115/volumes" Oct 29 22:35:59.777497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:59.776734 5367 generic.go:356] "Generic (PLEG): container finished" podID="78e29c1c-7b1e-4adf-970f-9a876045babb" containerID="8271b9130bc6e0af0a6b9997a77591b955c69f4656964dd579a3ceb7e2a89787" exitCode=0 Oct 29 22:35:59.777497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:59.776849 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" event={"ID":"78e29c1c-7b1e-4adf-970f-9a876045babb","Type":"ContainerDied","Data":"8271b9130bc6e0af0a6b9997a77591b955c69f4656964dd579a3ceb7e2a89787"} Oct 29 22:35:59.888224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:35:59.888199 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:36:00.003021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.002986 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003197 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003241 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003265 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:00.003568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003350 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003415 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003486 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003608 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003734 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:00.003778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003746 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket\") pod \"78e29c1c-7b1e-4adf-970f-9a876045babb\" (UID: \"78e29c1c-7b1e-4adf-970f-9a876045babb\") " Oct 29 22:36:00.003982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.003879 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:00.004429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.004402 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:00.004685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.004656 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:00.004826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.004804 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.004898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.004832 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e29c1c-7b1e-4adf-970f-9a876045babb-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.004898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.004848 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.004898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.004862 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.006880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.006848 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:00.007468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.006897 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:36:00.007468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.006998 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:00.007468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.007023 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj" pod "78e29c1c-7b1e-4adf-970f-9a876045babb" (UID: "78e29c1c-7b1e-4adf-970f-9a876045babb"). InnerVolumeSpecName "kube-api-access-w2twj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:00.106594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.106560 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.106594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.106590 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.106594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.106600 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e29c1c-7b1e-4adf-970f-9a876045babb-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.106861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.106609 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e29c1c-7b1e-4adf-970f-9a876045babb-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.106861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.106619 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2twj\" (UniqueName: \"kubernetes.io/projected/78e29c1c-7b1e-4adf-970f-9a876045babb-kube-api-access-w2twj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:00.218055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.217979 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj"] Oct 29 22:36:00.218897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218860 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e80875a1-4893-49d4-a5fe-32164c6e6efb" containerName="payload-processing" Oct 29 22:36:00.218897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218884 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80875a1-4893-49d4-a5fe-32164c6e6efb" containerName="payload-processing" Oct 29 22:36:00.219215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218931 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4f021dfe-de47-4e20-a024-30858ed4230b" containerName="payload-pre-processing" Oct 29 22:36:00.219215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218939 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f021dfe-de47-4e20-a024-30858ed4230b" containerName="payload-pre-processing" Oct 29 22:36:00.219215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218964 5367 cpu_manager.go:414] "containerMap: removing container" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" Oct 29 22:36:00.219215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218971 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" Oct 29 22:36:00.219215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218993 5367 cpu_manager.go:414] "containerMap: removing container" podUID="78e29c1c-7b1e-4adf-970f-9a876045babb" containerName="istio-proxy" Oct 29 22:36:00.219215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.218999 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e29c1c-7b1e-4adf-970f-9a876045babb" containerName="istio-proxy" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219232 5367 cpu_manager.go:414] "containerMap: removing container" podUID="408d80f0-2c15-4dbe-ad81-1f4f852c8004" containerName="revoke-keys" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219247 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408d80f0-2c15-4dbe-ad81-1f4f852c8004" containerName="revoke-keys" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219450 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="408d80f0-2c15-4dbe-ad81-1f4f852c8004" containerName="revoke-keys" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219464 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80875a1-4893-49d4-a5fe-32164c6e6efb" containerName="payload-processing" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219478 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f021dfe-de47-4e20-a024-30858ed4230b" containerName="payload-pre-processing" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219490 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="00116d89-c80a-409d-b395-27b4a2bb2b98" containerName="istio-proxy" Oct 29 22:36:00.219577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.219503 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e29c1c-7b1e-4adf-970f-9a876045babb" containerName="istio-proxy" Oct 29 22:36:00.222842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.222823 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.239269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.239247 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-36b46d\"" Oct 29 22:36:00.243525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.243500 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj"] Oct 29 22:36:00.281731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.281688 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887"] Oct 29 22:36:00.286661 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.286638 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.303566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.303542 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-36b46d-dockercfg-bms22\"" Oct 29 22:36:00.308942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.308913 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.309416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.309388 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tzq\" (UniqueName: \"kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq\") pod \"payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.311768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.311745 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887"] Oct 29 22:36:00.411689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.411632 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.411935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.411762 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsgj\" (UniqueName: \"kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj\") pod \"payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.411935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.411822 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.412115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.412098 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tzq\" (UniqueName: \"kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq\") pod \"payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.413387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.413358 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.433394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.433366 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tzq\" (UniqueName: \"kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq\") pod \"payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.523794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.514089 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsgj\" (UniqueName: \"kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj\") pod \"payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.523794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.514174 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.523794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.515439 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.536385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.536352 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsgj\" (UniqueName: \"kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj\") pod \"payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.536558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.536408 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:00.601568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.601539 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:00.794235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.794152 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" event={"ID":"78e29c1c-7b1e-4adf-970f-9a876045babb","Type":"ContainerDied","Data":"cfa51b3a5c99ecfb99317d3a00c7e0f2e0a44f569c0eddc9c52ea3d1d3a79b8f"} Oct 29 22:36:00.794235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.794224 5367 scope.go:122] "RemoveContainer" containerID="8271b9130bc6e0af0a6b9997a77591b955c69f4656964dd579a3ceb7e2a89787" Oct 29 22:36:00.798073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.795803 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz" Oct 29 22:36:00.858611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.858364 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj"] Oct 29 22:36:00.873299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.873271 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz"] Oct 29 22:36:00.880249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.880221 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-591986bb-openshift-default-cb894bd5-5thqz"] Oct 29 22:36:00.956085 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:00.956055 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa8264e_4f5e_4980_8cbb_8863df25b955.slice/crio-1fdc3b6bc4cfde8078cc78ed4bc8cb48e638e3dea98cbd5e088d8bfa8908f7e5 WatchSource:0}: Error finding container 1fdc3b6bc4cfde8078cc78ed4bc8cb48e638e3dea98cbd5e088d8bfa8908f7e5: Status 404 returned error can't find the container with id 1fdc3b6bc4cfde8078cc78ed4bc8cb48e638e3dea98cbd5e088d8bfa8908f7e5 Oct 29 22:36:00.959353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:00.959328 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887"] Oct 29 22:36:01.225848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.225815 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e29c1c-7b1e-4adf-970f-9a876045babb" path="/var/lib/kubelet/pods/78e29c1c-7b1e-4adf-970f-9a876045babb/volumes" Oct 29 22:36:01.810773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.810736 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" event={"ID":"5aa8264e-4f5e-4980-8cbb-8863df25b955","Type":"ContainerStarted","Data":"6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1"} Oct 29 22:36:01.811204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.810800 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" event={"ID":"5aa8264e-4f5e-4980-8cbb-8863df25b955","Type":"ContainerStarted","Data":"1fdc3b6bc4cfde8078cc78ed4bc8cb48e638e3dea98cbd5e088d8bfa8908f7e5"} Oct 29 22:36:01.811204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.810860 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:01.812437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.812416 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" event={"ID":"786ff7e6-15b5-4716-b6b5-ac7f306d3a14","Type":"ContainerStarted","Data":"7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51"} Oct 29 22:36:01.812535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.812449 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" event={"ID":"786ff7e6-15b5-4716-b6b5-ac7f306d3a14","Type":"ContainerStarted","Data":"981cdd658c6d4a8b35c1aeb03fe97d37f17326c03ea3f5a525251ffe749a445a"} Oct 29 22:36:01.812535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.812471 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:01.855041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.854991 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" podStartSLOduration=1.8549772679999998 podStartE2EDuration="1.854977268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:01.850733394 +0000 UTC m=+1228.744808315" watchObservedRunningTime="2026-10-29 22:36:01.854977268 +0000 UTC m=+1228.749052188" Oct 29 22:36:01.896550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:01.896495 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" podStartSLOduration=1.896481654 podStartE2EDuration="1.896481654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:01.892418283 +0000 UTC m=+1228.786493203" watchObservedRunningTime="2026-10-29 22:36:01.896481654 +0000 UTC m=+1228.790556572" Oct 29 22:36:02.197587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.197516 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd"] Oct 29 22:36:02.201282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.201266 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.221295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.221268 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:36:02.221947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.221911 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:36:02.231771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.231681 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:36:02.231936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.231868 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:36:02.243516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.243484 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd"] Oct 29 22:36:02.257128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.257091 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.257280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.257215 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.257360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.257341 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxgvk\" (UniqueName: \"kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.358232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.358199 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.358415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.358308 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxgvk\" (UniqueName: \"kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.358569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.358549 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.358975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.358954 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.360842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.360812 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.383570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.383548 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxgvk\" (UniqueName: \"kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk\") pod \"maas-api-revoke-sub-ac561c2c-qztxd\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.538771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.536574 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:02.859635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:02.859603 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd"] Oct 29 22:36:03.030955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:03.030919 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz"] Oct 29 22:36:03.036349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:03.036324 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647a4e3f-gk7bz"] Oct 29 22:36:03.223930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:03.223903 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52566829-6e5d-4a4f-8777-ce55ab488111" path="/var/lib/kubelet/pods/52566829-6e5d-4a4f-8777-ce55ab488111/volumes" Oct 29 22:36:03.837305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:03.837271 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" event={"ID":"1f825084-5ce2-447d-ace5-80c875e3b95b","Type":"ContainerStarted","Data":"2b3b1d82f8500ae2de78becfcb4ad45b33add40a317a5334b858bfbb99b3ca32"} Oct 29 22:36:03.837499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:03.837319 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" event={"ID":"1f825084-5ce2-447d-ace5-80c875e3b95b","Type":"ContainerStarted","Data":"88e155050e917fd8d3c8ac6271c29bd9a5baf627e041b5d58a4716d7784e11e0"} Oct 29 22:36:03.878640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:03.878592 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" podStartSLOduration=1.8785795379999999 podStartE2EDuration="1.878579538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:03.876661566 +0000 UTC m=+1230.770736486" watchObservedRunningTime="2026-10-29 22:36:03.878579538 +0000 UTC m=+1230.772654487" Oct 29 22:36:04.848019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:04.847985 5367 generic.go:356] "Generic (PLEG): container finished" podID="1f825084-5ce2-447d-ace5-80c875e3b95b" containerID="2b3b1d82f8500ae2de78becfcb4ad45b33add40a317a5334b858bfbb99b3ca32" exitCode=0 Oct 29 22:36:04.848225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:04.848070 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" event={"ID":"1f825084-5ce2-447d-ace5-80c875e3b95b","Type":"ContainerDied","Data":"2b3b1d82f8500ae2de78becfcb4ad45b33add40a317a5334b858bfbb99b3ca32"} Oct 29 22:36:06.018356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.018310 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:06.031517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.031491 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4"] Oct 29 22:36:06.045170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.045142 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff38a7d0-8kxh4"] Oct 29 22:36:06.122894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.122815 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca\") pod \"1f825084-5ce2-447d-ace5-80c875e3b95b\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " Oct 29 22:36:06.123058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.122912 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca\") pod \"1f825084-5ce2-447d-ace5-80c875e3b95b\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " Oct 29 22:36:06.123244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.123218 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk\" (UniqueName: \"kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk\") pod \"1f825084-5ce2-447d-ace5-80c875e3b95b\" (UID: \"1f825084-5ce2-447d-ace5-80c875e3b95b\") " Oct 29 22:36:06.123385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.123277 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca" pod "1f825084-5ce2-447d-ace5-80c875e3b95b" (UID: "1f825084-5ce2-447d-ace5-80c875e3b95b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:06.123848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.123829 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:06.125017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.124996 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca" pod "1f825084-5ce2-447d-ace5-80c875e3b95b" (UID: "1f825084-5ce2-447d-ace5-80c875e3b95b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:06.125207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.125195 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk" pod "1f825084-5ce2-447d-ace5-80c875e3b95b" (UID: "1f825084-5ce2-447d-ace5-80c875e3b95b"). InnerVolumeSpecName "kube-api-access-xxgvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:06.224882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.224854 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxgvk\" (UniqueName: \"kubernetes.io/projected/1f825084-5ce2-447d-ace5-80c875e3b95b-kube-api-access-xxgvk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:06.224882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.224881 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f825084-5ce2-447d-ace5-80c875e3b95b-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:06.870436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.870398 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" Oct 29 22:36:06.870662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.870433 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd" event={"ID":"1f825084-5ce2-447d-ace5-80c875e3b95b","Type":"ContainerDied","Data":"88e155050e917fd8d3c8ac6271c29bd9a5baf627e041b5d58a4716d7784e11e0"} Oct 29 22:36:06.870662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:06.870506 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e155050e917fd8d3c8ac6271c29bd9a5baf627e041b5d58a4716d7784e11e0" Oct 29 22:36:07.224837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:07.224758 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791d6ae9-d240-49e3-aa64-5b71f7287493" path="/var/lib/kubelet/pods/791d6ae9-d240-49e3-aa64-5b71f7287493/volumes" Oct 29 22:36:12.829179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:12.829151 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 22:36:12.829586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:12.829238 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 22:36:14.353930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.353897 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl"] Oct 29 22:36:14.354678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.354663 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1f825084-5ce2-447d-ace5-80c875e3b95b" containerName="revoke-keys" Oct 29 22:36:14.354760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.354680 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f825084-5ce2-447d-ace5-80c875e3b95b" containerName="revoke-keys" Oct 29 22:36:14.354853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.354842 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f825084-5ce2-447d-ace5-80c875e3b95b" containerName="revoke-keys" Oct 29 22:36:14.358276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.358255 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.379333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.379305 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl"] Oct 29 22:36:14.386022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.386001 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:36:14.386154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.386025 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:36:14.386154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.385967 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:36:14.387359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.387337 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:36:14.452619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.452585 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.452857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.452837 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkj9t\" (UniqueName: \"kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.452919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.452868 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.554900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.554862 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkj9t\" (UniqueName: \"kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.555098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.554910 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.555169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.555104 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.555592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.555569 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.557373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.557354 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.574992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.574964 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkj9t\" (UniqueName: \"kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t\") pod \"maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.671540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.671456 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:14.977990 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:14.977960 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78a5a32_c5e5_4a08_8a17_3a5c29f9d341.slice/crio-6957aea65e5d2c2367fb8a5805838dfb2b7fb006e3e112c59400c6f70194b042 WatchSource:0}: Error finding container 6957aea65e5d2c2367fb8a5805838dfb2b7fb006e3e112c59400c6f70194b042: Status 404 returned error can't find the container with id 6957aea65e5d2c2367fb8a5805838dfb2b7fb006e3e112c59400c6f70194b042 Oct 29 22:36:14.980158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:14.980133 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl"] Oct 29 22:36:15.978303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:15.978260 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" event={"ID":"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341","Type":"ContainerStarted","Data":"14d3363a45d6f018aeddf8f2b37445f6ef9f14f06c8a3d9baba50b90fc7989ab"} Oct 29 22:36:15.978799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:15.978336 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" event={"ID":"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341","Type":"ContainerStarted","Data":"6957aea65e5d2c2367fb8a5805838dfb2b7fb006e3e112c59400c6f70194b042"} Oct 29 22:36:16.020043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:16.019983 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" podStartSLOduration=2.019961713 podStartE2EDuration="2.019961713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:16.017509851 +0000 UTC m=+1242.911584772" watchObservedRunningTime="2026-10-29 22:36:16.019961713 +0000 UTC m=+1242.914036634" Oct 29 22:36:16.991003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:16.990963 5367 generic.go:356] "Generic (PLEG): container finished" podID="f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" containerID="14d3363a45d6f018aeddf8f2b37445f6ef9f14f06c8a3d9baba50b90fc7989ab" exitCode=0 Oct 29 22:36:16.991452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:16.991026 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" event={"ID":"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341","Type":"ContainerDied","Data":"14d3363a45d6f018aeddf8f2b37445f6ef9f14f06c8a3d9baba50b90fc7989ab"} Oct 29 22:36:18.168739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.168686 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:18.214670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.214636 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca\") pod \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " Oct 29 22:36:18.214951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.214928 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca\") pod \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " Oct 29 22:36:18.215112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.215094 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t\" (UniqueName: \"kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t\") pod \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\" (UID: \"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341\") " Oct 29 22:36:18.215799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.215334 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca" pod "f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" (UID: "f78a5a32-c5e5-4a08-8a17-3a5c29f9d341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:18.215893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.215813 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:18.217243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.217216 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca" pod "f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" (UID: "f78a5a32-c5e5-4a08-8a17-3a5c29f9d341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:18.217444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.217426 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t" pod "f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" (UID: "f78a5a32-c5e5-4a08-8a17-3a5c29f9d341"). InnerVolumeSpecName "kube-api-access-pkj9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:18.317404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.317372 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkj9t\" (UniqueName: \"kubernetes.io/projected/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-kube-api-access-pkj9t\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:18.317404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:18.317400 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:19.014300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:19.014269 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" Oct 29 22:36:19.014300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:19.014286 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl" event={"ID":"f78a5a32-c5e5-4a08-8a17-3a5c29f9d341","Type":"ContainerDied","Data":"6957aea65e5d2c2367fb8a5805838dfb2b7fb006e3e112c59400c6f70194b042"} Oct 29 22:36:19.014539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:19.014334 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6957aea65e5d2c2367fb8a5805838dfb2b7fb006e3e112c59400c6f70194b042" Oct 29 22:36:22.332414 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:36:22.332378 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:36:24.780613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:24.780579 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b"] Oct 29 22:36:26.033151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.033112 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw"] Oct 29 22:36:26.043830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.043794 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b857fc-wp9bw"] Oct 29 22:36:26.934258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.934224 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll"] Oct 29 22:36:26.935395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.935373 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" containerName="revoke-keys" Oct 29 22:36:26.935496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.935397 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" containerName="revoke-keys" Oct 29 22:36:26.935613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.935602 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" containerName="revoke-keys" Oct 29 22:36:26.940191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.940170 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:26.957455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.957432 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-afdbe8-openshift-default-dockercfg-crwqv\"" Oct 29 22:36:26.981634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:26.981139 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll"] Oct 29 22:36:27.061773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.061738 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.061817 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.061857 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.062078 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.062105 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.062176 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p56m\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.062225 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.062309 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.062415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.062355 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.163494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163459 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.163690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163543 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.163690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163583 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.163853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163834 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.163910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163867 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.163971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163914 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p56m\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.163967 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164009 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164016 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164089 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164197 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164224 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164334 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.164577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.164557 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.166962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.166940 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.167075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.167056 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.185102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.185048 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.185468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.185452 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p56m\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.254729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.230946 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b251e382-ed22-417c-a854-2e20d97f35c4" path="/var/lib/kubelet/pods/b251e382-ed22-417c-a854-2e20d97f35c4/volumes" Oct 29 22:36:27.276737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.267055 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:27.672376 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:27.672342 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4887aad4_4623_463e_a66c_f470439dc173.slice/crio-03b56ad1be2fa0531976f73481768ab138ab08142b9ffe8093d8a37ab382efd0 WatchSource:0}: Error finding container 03b56ad1be2fa0531976f73481768ab138ab08142b9ffe8093d8a37ab382efd0: Status 404 returned error can't find the container with id 03b56ad1be2fa0531976f73481768ab138ab08142b9ffe8093d8a37ab382efd0 Oct 29 22:36:27.673437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.673407 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll"] Oct 29 22:36:27.675779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.675737 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:27.675848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.675825 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:27.675882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:27.675871 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:28.115383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:28.115342 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" event={"ID":"4887aad4-4623-463e-a66c-f470439dc173","Type":"ContainerStarted","Data":"9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4"} Oct 29 22:36:28.115836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:28.115404 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" event={"ID":"4887aad4-4623-463e-a66c-f470439dc173","Type":"ContainerStarted","Data":"03b56ad1be2fa0531976f73481768ab138ab08142b9ffe8093d8a37ab382efd0"} Oct 29 22:36:28.175870 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:28.175787 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podStartSLOduration=2.175769547 podStartE2EDuration="2.175769547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:28.172420484 +0000 UTC m=+1255.066495428" watchObservedRunningTime="2026-10-29 22:36:28.175769547 +0000 UTC m=+1255.069844479" Oct 29 22:36:28.270588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:28.270553 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:28.276457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:28.276432 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:29.125222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:29.125192 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:29.126080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:29.126063 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:36:29.781958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:29.781919 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" podUID="b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" containerName="payload-processing" containerID="cri-o://8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1" gracePeriod=25 Oct 29 22:36:30.072873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.072845 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:36:30.138655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.138618 5367 generic.go:356] "Generic (PLEG): container finished" podID="b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" containerID="8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1" exitCode=0 Oct 29 22:36:30.139163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.138833 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" Oct 29 22:36:30.139163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.138850 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" event={"ID":"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8","Type":"ContainerDied","Data":"8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1"} Oct 29 22:36:30.139163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.138912 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b" event={"ID":"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8","Type":"ContainerDied","Data":"bde4296596d76ff0856410990d7ffc6b76ceb9c07b54fe63d493ac011b718cb7"} Oct 29 22:36:30.139350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.139334 5367 scope.go:122] "RemoveContainer" containerID="8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1" Oct 29 22:36:30.151732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.151712 5367 scope.go:122] "RemoveContainer" containerID="8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1" Oct 29 22:36:30.152074 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:36:30.152046 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1\": container with ID starting with 8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1 not found: ID does not exist" containerID="8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1" Oct 29 22:36:30.152179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.152076 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1"} err="failed to get container status \"8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1\": rpc error: code = NotFound desc = could not find container \"8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1\": container with ID starting with 8a6659abff3b6b38211ddfd20f8e6f3c4b46e0470f26ba94fd0eaf6ca544dae1 not found: ID does not exist" Oct 29 22:36:30.227109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.227077 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume\") pod \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " Oct 29 22:36:30.227109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.227110 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj\" (UniqueName: \"kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj\") pod \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\" (UID: \"b72be0a4-5ae6-4dbe-b265-dcc20aab98b8\") " Oct 29 22:36:30.227502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.227478 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume" pod "b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" (UID: "b72be0a4-5ae6-4dbe-b265-dcc20aab98b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:30.227726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.227676 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:30.229208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.229180 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj" pod "b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" (UID: "b72be0a4-5ae6-4dbe-b265-dcc20aab98b8"). InnerVolumeSpecName "kube-api-access-h7glj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:30.329793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.329755 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7glj\" (UniqueName: \"kubernetes.io/projected/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8-kube-api-access-h7glj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:30.489903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.489863 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b"] Oct 29 22:36:30.495631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:30.495599 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1d9591d-7c6c8cb89d-fb88b"] Oct 29 22:36:31.224347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:31.224314 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" path="/var/lib/kubelet/pods/b72be0a4-5ae6-4dbe-b265-dcc20aab98b8/volumes" Oct 29 22:36:32.386020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.385975 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj"] Oct 29 22:36:32.386775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.386756 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" containerName="payload-processing" Oct 29 22:36:32.386825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.386776 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" containerName="payload-processing" Oct 29 22:36:32.386902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.386893 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72be0a4-5ae6-4dbe-b265-dcc20aab98b8" containerName="payload-processing" Oct 29 22:36:32.392341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.392311 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.418023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.417990 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj"] Oct 29 22:36:32.420955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.420932 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-afdbe8\"" Oct 29 22:36:32.471722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.465444 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.471722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.465577 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tjr\" (UniqueName: \"kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr\") pod \"payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.535770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.529788 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-afdbe8-dockercfg-ws9h7\"" Oct 29 22:36:32.566894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.566855 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tjr\" (UniqueName: \"kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr\") pod \"payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.567159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.567134 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.568023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.567997 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.590508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.590474 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tjr\" (UniqueName: \"kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr\") pod \"payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:32.711769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:32.710469 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:33.018217 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:33.018180 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d2ad78_ba6d_4c83_881e_54296aff2673.slice/crio-6ab0e372c669a526e8c487dc7b131b4b0bd2828ce5b04898e3476b34fe727fc4 WatchSource:0}: Error finding container 6ab0e372c669a526e8c487dc7b131b4b0bd2828ce5b04898e3476b34fe727fc4: Status 404 returned error can't find the container with id 6ab0e372c669a526e8c487dc7b131b4b0bd2828ce5b04898e3476b34fe727fc4 Oct 29 22:36:33.020007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:33.019925 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj"] Oct 29 22:36:33.177585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:33.177548 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" event={"ID":"d3d2ad78-ba6d-4c83-881e-54296aff2673","Type":"ContainerStarted","Data":"db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d"} Oct 29 22:36:33.177775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:33.177604 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" event={"ID":"d3d2ad78-ba6d-4c83-881e-54296aff2673","Type":"ContainerStarted","Data":"6ab0e372c669a526e8c487dc7b131b4b0bd2828ce5b04898e3476b34fe727fc4"} Oct 29 22:36:33.177775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:33.177643 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:33.227280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:33.227221 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" podStartSLOduration=1.227202711 podStartE2EDuration="1.227202711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:33.219214568 +0000 UTC m=+1260.113289487" watchObservedRunningTime="2026-10-29 22:36:33.227202711 +0000 UTC m=+1260.121277634" Oct 29 22:36:33.989129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:33.989092 5367 scope.go:122] "RemoveContainer" containerID="d7ee54b96677b66e8333036656ee7caf612f0a24f974959c58fefab4d0d58366" Oct 29 22:36:34.000708 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.000677 5367 scope.go:122] "RemoveContainer" containerID="82d4420d6a0ce3de80acf858afe741fccb1b1cda92bf2125efb9639e403d66c2" Oct 29 22:36:34.011256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.011237 5367 scope.go:122] "RemoveContainer" containerID="073e3b3a3d87a5218d3a43d60f53f74616e6314895cbda95fda6b5c23250184a" Oct 29 22:36:34.021869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.021853 5367 scope.go:122] "RemoveContainer" containerID="b3476cdb53e1c9af2df6a22be348944dbfe615bfd8666a57b2b6f8bb26c12fbd" Oct 29 22:36:34.032777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.032760 5367 scope.go:122] "RemoveContainer" containerID="63f10639558dc00d25920faa7e68b780a4a918071bf3face9a94a307d6375f61" Oct 29 22:36:34.043316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.043301 5367 scope.go:122] "RemoveContainer" containerID="7777472665e2bb468af6492d4dd820c30e0540fc4ee68f68748b9ace73c19285" Oct 29 22:36:34.057407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.057389 5367 scope.go:122] "RemoveContainer" containerID="edfbe109ec0a3785e081f6bbc272b7859c268a1de4227826cd252b61a06631ea" Oct 29 22:36:34.067994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.067974 5367 scope.go:122] "RemoveContainer" containerID="81e01cc495b5296a8ebf10ca5ca6dc5d5c13d16dbead5309afd9f9f58e2dde9d" Oct 29 22:36:34.077370 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.077354 5367 scope.go:122] "RemoveContainer" containerID="fc0cd451ebd471630f00e5a6fee92c4fcdb233bd0a78c63d734959e50f8f4f1e" Oct 29 22:36:34.087442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.087424 5367 scope.go:122] "RemoveContainer" containerID="ce750159cd61be4a632f37dbdf4cd99ed0efe59ea44919d1491c04b093122ff0" Oct 29 22:36:34.097798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.097779 5367 scope.go:122] "RemoveContainer" containerID="dc1e27dddb2202083eeb02698296d925c862fc7b263bd9e9a80194d9552da4ce" Oct 29 22:36:34.108222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:34.108205 5367 scope.go:122] "RemoveContainer" containerID="f0a4762dca6c2d2bdb09d17d44782b06e2dbabd030ebc27130382f6bd233b4c2" Oct 29 22:36:44.199022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:44.198949 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 22:36:46.845731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:46.845676 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7"] Oct 29 22:36:46.896723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:46.880216 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7"] Oct 29 22:36:46.896723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:46.880329 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:46.963897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:46.929585 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8f1627\"" Oct 29 22:36:46.963897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:46.930261 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8f1627-dockercfg-cxq7n\"" Oct 29 22:36:47.068723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.060571 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz"] Oct 29 22:36:47.077591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.069841 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz"] Oct 29 22:36:47.077591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.069973 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.163721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.156011 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfq7\" (UniqueName: \"kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7\") pod \"payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.163721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.156157 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9j76\" (UniqueName: \"kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76\") pod \"payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.163721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.156309 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.163721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.156357 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.258837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.258799 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.260460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.260439 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.261716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.261681 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfq7\" (UniqueName: \"kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7\") pod \"payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.262286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.262266 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9j76\" (UniqueName: \"kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76\") pod \"payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.263154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.260371 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.263298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.261396 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.333720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.332835 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfq7\" (UniqueName: \"kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7\") pod \"payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.349526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.346176 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9j76\" (UniqueName: \"kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76\") pod \"payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.565191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.565164 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:47.613161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.613135 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:47.886210 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:47.886143 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3136ce3f_d29d_4161_be50_ed68a4fc8cb4.slice/crio-d904773a86ebbc0dfae004fbbd3b4b8f1dacc0bf9992627e5a13069ea0c7ae7d WatchSource:0}: Error finding container d904773a86ebbc0dfae004fbbd3b4b8f1dacc0bf9992627e5a13069ea0c7ae7d: Status 404 returned error can't find the container with id d904773a86ebbc0dfae004fbbd3b4b8f1dacc0bf9992627e5a13069ea0c7ae7d Oct 29 22:36:47.888080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.888058 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz"] Oct 29 22:36:47.929953 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:47.929923 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08968dc4_9f0b_4776_b957_b8835c134819.slice/crio-b67dea07144905684801759ef2559a8c73995e7b5cfe1a885ecb2ee7f363b178 WatchSource:0}: Error finding container b67dea07144905684801759ef2559a8c73995e7b5cfe1a885ecb2ee7f363b178: Status 404 returned error can't find the container with id b67dea07144905684801759ef2559a8c73995e7b5cfe1a885ecb2ee7f363b178 Oct 29 22:36:47.931855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:47.931833 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7"] Oct 29 22:36:48.476924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.476881 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" event={"ID":"08968dc4-9f0b-4776-b957-b8835c134819","Type":"ContainerStarted","Data":"a41ba8297796640ed4535cfafab76591189f2cd8f7f2aa2e646769906f1a3ccc"} Oct 29 22:36:48.477129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.476980 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:36:48.477129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.477003 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" event={"ID":"08968dc4-9f0b-4776-b957-b8835c134819","Type":"ContainerStarted","Data":"b67dea07144905684801759ef2559a8c73995e7b5cfe1a885ecb2ee7f363b178"} Oct 29 22:36:48.478849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.478808 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" event={"ID":"3136ce3f-d29d-4161-be50-ed68a4fc8cb4","Type":"ContainerStarted","Data":"8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09"} Oct 29 22:36:48.478984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.478866 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" event={"ID":"3136ce3f-d29d-4161-be50-ed68a4fc8cb4","Type":"ContainerStarted","Data":"d904773a86ebbc0dfae004fbbd3b4b8f1dacc0bf9992627e5a13069ea0c7ae7d"} Oct 29 22:36:48.478984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.478931 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:48.520085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.520042 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" podStartSLOduration=2.520028569 podStartE2EDuration="2.520028569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:48.516249294 +0000 UTC m=+1275.410324214" watchObservedRunningTime="2026-10-29 22:36:48.520028569 +0000 UTC m=+1275.414103488" Oct 29 22:36:48.560802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:48.560752 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" podStartSLOduration=2.560739746 podStartE2EDuration="2.560739746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:48.556146934 +0000 UTC m=+1275.450221855" watchObservedRunningTime="2026-10-29 22:36:48.560739746 +0000 UTC m=+1275.454814661" Oct 29 22:36:50.611440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.611405 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl"] Oct 29 22:36:50.624403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.624369 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1d9591d-5dpvl"] Oct 29 22:36:50.719292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.719258 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq"] Oct 29 22:36:50.724224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.724202 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.742433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.742388 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq"] Oct 29 22:36:50.742680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.742534 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:36:50.742680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.742604 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:36:50.819590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.819563 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:36:50.819801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.819648 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:36:50.822670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.822648 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.822956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.822938 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.823023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.823007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph6jj\" (UniqueName: \"kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.924211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.924126 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.924371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.924216 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph6jj\" (UniqueName: \"kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.924371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.924319 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.924891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.924871 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.926745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.926683 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:50.945247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:50.945216 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph6jj\" (UniqueName: \"kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj\") pod \"maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:51.050061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:51.050030 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:51.233039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:51.232950 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78a5a32-c5e5-4a08-8a17-3a5c29f9d341" path="/var/lib/kubelet/pods/f78a5a32-c5e5-4a08-8a17-3a5c29f9d341/volumes" Oct 29 22:36:51.366648 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:51.366616 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b57ffe3_7afa_421f_b8f2_330b43a71353.slice/crio-083930a9769abaf7985f077b9f8d6318a5ff0295667be8a64edabf26e50eae5a WatchSource:0}: Error finding container 083930a9769abaf7985f077b9f8d6318a5ff0295667be8a64edabf26e50eae5a: Status 404 returned error can't find the container with id 083930a9769abaf7985f077b9f8d6318a5ff0295667be8a64edabf26e50eae5a Oct 29 22:36:51.367247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:51.367226 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq"] Oct 29 22:36:51.511818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:51.511728 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" event={"ID":"0b57ffe3-7afa-421f-b8f2-330b43a71353","Type":"ContainerStarted","Data":"083930a9769abaf7985f077b9f8d6318a5ff0295667be8a64edabf26e50eae5a"} Oct 29 22:36:52.029609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:52.029525 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb"] Oct 29 22:36:52.043052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:52.043019 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ed72aa-gkxdb"] Oct 29 22:36:52.356141 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:36:52.356075 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:36:52.538768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:52.527646 5367 generic.go:356] "Generic (PLEG): container finished" podID="0b57ffe3-7afa-421f-b8f2-330b43a71353" containerID="bb857e2a51acb41eb70d6228bb1c359768c3a60da1031352423da014536876b6" exitCode=0 Oct 29 22:36:52.538768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:52.527749 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" event={"ID":"0b57ffe3-7afa-421f-b8f2-330b43a71353","Type":"ContainerDied","Data":"bb857e2a51acb41eb70d6228bb1c359768c3a60da1031352423da014536876b6"} Oct 29 22:36:53.223878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.223847 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263ffebb-1be0-4998-a15d-1b1477faf5ec" path="/var/lib/kubelet/pods/263ffebb-1be0-4998-a15d-1b1477faf5ec/volumes" Oct 29 22:36:53.359325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.359157 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv"] Oct 29 22:36:53.364259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.364235 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.380337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.380312 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-e81b56-openshift-default-dockercfg-cgfcs\"" Oct 29 22:36:53.402548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.402509 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv"] Oct 29 22:36:53.468216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468180 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468293 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468433 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468539 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468563 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqvz\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468594 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468668 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468782 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.468835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.468828 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.570693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.570648 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.570693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.570723 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqvz\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.570759 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.570802 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.570877 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.570924 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571105 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571162 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571209 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571396 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571596 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.571955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.571933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.572464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.572436 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.576647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.576067 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.576647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.576291 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.624414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.622356 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.641159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.635214 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqvz\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.700804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.691056 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:53.786873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.786146 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:53.881664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.881631 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj\" (UniqueName: \"kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj\") pod \"0b57ffe3-7afa-421f-b8f2-330b43a71353\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " Oct 29 22:36:53.881879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.881683 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca\") pod \"0b57ffe3-7afa-421f-b8f2-330b43a71353\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " Oct 29 22:36:53.881957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.881885 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca\") pod \"0b57ffe3-7afa-421f-b8f2-330b43a71353\" (UID: \"0b57ffe3-7afa-421f-b8f2-330b43a71353\") " Oct 29 22:36:53.882226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.882202 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca" pod "0b57ffe3-7afa-421f-b8f2-330b43a71353" (UID: "0b57ffe3-7afa-421f-b8f2-330b43a71353"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:53.882666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.882647 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:53.883849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.883827 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca" pod "0b57ffe3-7afa-421f-b8f2-330b43a71353" (UID: "0b57ffe3-7afa-421f-b8f2-330b43a71353"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:53.883960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.883945 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj" pod "0b57ffe3-7afa-421f-b8f2-330b43a71353" (UID: "0b57ffe3-7afa-421f-b8f2-330b43a71353"). InnerVolumeSpecName "kube-api-access-ph6jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:53.987275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.987243 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph6jj\" (UniqueName: \"kubernetes.io/projected/0b57ffe3-7afa-421f-b8f2-330b43a71353-kube-api-access-ph6jj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:53.987275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:53.987270 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b57ffe3-7afa-421f-b8f2-330b43a71353-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:36:54.053938 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:36:54.053904 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e312dd8_3618_4b93_9fc6_321f2460aa77.slice/crio-d3d465ef6f491df70968fbf7124c55781e44b57e0be47a70b80532fe7a9d4216 WatchSource:0}: Error finding container d3d465ef6f491df70968fbf7124c55781e44b57e0be47a70b80532fe7a9d4216: Status 404 returned error can't find the container with id d3d465ef6f491df70968fbf7124c55781e44b57e0be47a70b80532fe7a9d4216 Oct 29 22:36:54.055933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.055898 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv"] Oct 29 22:36:54.058094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.058041 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:54.058217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.058129 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:54.058217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.058171 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:54.547366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.547338 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" Oct 29 22:36:54.547366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.547351 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq" event={"ID":"0b57ffe3-7afa-421f-b8f2-330b43a71353","Type":"ContainerDied","Data":"083930a9769abaf7985f077b9f8d6318a5ff0295667be8a64edabf26e50eae5a"} Oct 29 22:36:54.547882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.547408 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="083930a9769abaf7985f077b9f8d6318a5ff0295667be8a64edabf26e50eae5a" Oct 29 22:36:54.549382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.549359 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" event={"ID":"2e312dd8-3618-4b93-9fc6-321f2460aa77","Type":"ContainerStarted","Data":"b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71"} Oct 29 22:36:54.549494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.549399 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" event={"ID":"2e312dd8-3618-4b93-9fc6-321f2460aa77","Type":"ContainerStarted","Data":"d3d465ef6f491df70968fbf7124c55781e44b57e0be47a70b80532fe7a9d4216"} Oct 29 22:36:54.615495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.615386 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podStartSLOduration=1.6153698539999999 podStartE2EDuration="1.615369854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:54.614113641 +0000 UTC m=+1281.508188561" watchObservedRunningTime="2026-10-29 22:36:54.615369854 +0000 UTC m=+1281.509444777" Oct 29 22:36:54.692520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:54.692480 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:55.699350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:55.699319 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:56.570888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:56.570848 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:56.571999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:56.571977 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:36:59.490675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:59.490644 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 22:36:59.491073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:36:59.491016 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 22:37:01.144909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:01.144873 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s"] Oct 29 22:37:01.168523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:01.168491 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5"] Oct 29 22:37:06.145846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.145801 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" podUID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" containerName="payload-processing" containerID="cri-o://0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2" gracePeriod=25 Oct 29 22:37:06.169004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.168942 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" podUID="7c9579b0-9d8f-4caa-babc-69ba5a02a012" containerName="payload-pre-processing" containerID="cri-o://dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f" gracePeriod=25 Oct 29 22:37:06.575130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.575105 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:37:06.660846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.659672 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume\") pod \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " Oct 29 22:37:06.660846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.659863 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz\" (UniqueName: \"kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz\") pod \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\" (UID: \"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb\") " Oct 29 22:37:06.660846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.660125 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume" pod "06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" (UID: "06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:06.660846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.660558 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.662521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.662501 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz" pod "06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" (UID: "06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb"). InnerVolumeSpecName "kube-api-access-jwqhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:06.698913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.696275 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:37:06.715740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.715691 5367 generic.go:356] "Generic (PLEG): container finished" podID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" containerID="0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2" exitCode=0 Oct 29 22:37:06.715938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.715751 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" event={"ID":"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb","Type":"ContainerDied","Data":"0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2"} Oct 29 22:37:06.715938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.715777 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" Oct 29 22:37:06.715938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.715823 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s" event={"ID":"06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb","Type":"ContainerDied","Data":"92220d90a71377bcd5d827a15b9fed7ba3d9763763331de829b01fa96e2a6b96"} Oct 29 22:37:06.715938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.715852 5367 scope.go:122] "RemoveContainer" containerID="0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2" Oct 29 22:37:06.721829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.721803 5367 generic.go:356] "Generic (PLEG): container finished" podID="7c9579b0-9d8f-4caa-babc-69ba5a02a012" containerID="dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f" exitCode=0 Oct 29 22:37:06.722076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.722053 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" event={"ID":"7c9579b0-9d8f-4caa-babc-69ba5a02a012","Type":"ContainerDied","Data":"dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f"} Oct 29 22:37:06.722152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.722109 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" event={"ID":"7c9579b0-9d8f-4caa-babc-69ba5a02a012","Type":"ContainerDied","Data":"dae6a914aadcb4def850d0fdcbf4b8660c2a4f34ba1ea40975dc8f01272bccc6"} Oct 29 22:37:06.722215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.722181 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5" Oct 29 22:37:06.750879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.750854 5367 scope.go:122] "RemoveContainer" containerID="0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2" Oct 29 22:37:06.751231 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:37:06.751208 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2\": container with ID starting with 0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2 not found: ID does not exist" containerID="0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2" Oct 29 22:37:06.751307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.751244 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2"} err="failed to get container status \"0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2\": rpc error: code = NotFound desc = could not find container \"0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2\": container with ID starting with 0eedf368ad629779837aed5e5fd6e8d24fa409cd69852a89806bcac34e4990f2 not found: ID does not exist" Oct 29 22:37:06.751307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.751270 5367 scope.go:122] "RemoveContainer" containerID="dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f" Oct 29 22:37:06.761356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.761326 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume\") pod \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " Oct 29 22:37:06.761504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.761488 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66\" (UniqueName: \"kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66\") pod \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\" (UID: \"7c9579b0-9d8f-4caa-babc-69ba5a02a012\") " Oct 29 22:37:06.761785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.761741 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume" pod "7c9579b0-9d8f-4caa-babc-69ba5a02a012" (UID: "7c9579b0-9d8f-4caa-babc-69ba5a02a012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:06.762272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.762248 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwqhz\" (UniqueName: \"kubernetes.io/projected/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb-kube-api-access-jwqhz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.762361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.762279 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9579b0-9d8f-4caa-babc-69ba5a02a012-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.770525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.770492 5367 scope.go:122] "RemoveContainer" containerID="dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f" Oct 29 22:37:06.771313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.771243 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66" pod "7c9579b0-9d8f-4caa-babc-69ba5a02a012" (UID: "7c9579b0-9d8f-4caa-babc-69ba5a02a012"). InnerVolumeSpecName "kube-api-access-mvn66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:06.782238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.782213 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s"] Oct 29 22:37:06.782442 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:37:06.782406 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f\": container with ID starting with dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f not found: ID does not exist" containerID="dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f" Oct 29 22:37:06.782509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.782472 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f"} err="failed to get container status \"dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f\": rpc error: code = NotFound desc = could not find container \"dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f\": container with ID starting with dffd83ecb4c290710abf8650d113637dbd1417c7fea76f2e3049c638b3e1144f not found: ID does not exist" Oct 29 22:37:06.788487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.788467 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f922c6e9-55f64bd4b8-kkk9s"] Oct 29 22:37:06.864284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:06.864250 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvn66\" (UniqueName: \"kubernetes.io/projected/7c9579b0-9d8f-4caa-babc-69ba5a02a012-kube-api-access-mvn66\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:37:07.020157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.020025 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq"] Oct 29 22:37:07.027319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.027289 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f922c6e9-7zcwq"] Oct 29 22:37:07.074323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.074287 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5"] Oct 29 22:37:07.080398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.080365 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f922c6e9-6854bb686c-n7lx5"] Oct 29 22:37:07.224906 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.224874 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" path="/var/lib/kubelet/pods/06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb/volumes" Oct 29 22:37:07.225515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.225499 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b57ffe3-7afa-421f-b8f2-330b43a71353" path="/var/lib/kubelet/pods/0b57ffe3-7afa-421f-b8f2-330b43a71353/volumes" Oct 29 22:37:07.225993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:07.225977 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9579b0-9d8f-4caa-babc-69ba5a02a012" path="/var/lib/kubelet/pods/7c9579b0-9d8f-4caa-babc-69ba5a02a012/volumes" Oct 29 22:37:12.035285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:12.035249 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl"] Oct 29 22:37:12.048351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:12.048319 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358c33ed-k2jnl"] Oct 29 22:37:13.225346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:13.225314 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9728a3-eefe-42db-bd84-e437b37ef405" path="/var/lib/kubelet/pods/da9728a3-eefe-42db-bd84-e437b37ef405/volumes" Oct 29 22:37:16.039758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:16.038625 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz"] Oct 29 22:37:16.047546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:16.047516 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c66bd32-dttvz"] Oct 29 22:37:17.224596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:17.224561 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e36897f-7dc0-484f-bf6c-81f1a960c428" path="/var/lib/kubelet/pods/4e36897f-7dc0-484f-bf6c-81f1a960c428/volumes" Oct 29 22:37:18.030547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:18.030512 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd"] Oct 29 22:37:18.043518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:18.043480 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa563490-shscd"] Oct 29 22:37:19.225124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:19.225083 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0fb474-5b3b-450f-8f1e-697e64bf9b70" path="/var/lib/kubelet/pods/7f0fb474-5b3b-450f-8f1e-697e64bf9b70/volumes" Oct 29 22:37:22.334202 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:37:22.333980 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:37:22.361977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.361947 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf"] Oct 29 22:37:22.362686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362671 5367 cpu_manager.go:414] "containerMap: removing container" podUID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" containerName="payload-processing" Oct 29 22:37:22.362744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362687 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" containerName="payload-processing" Oct 29 22:37:22.362744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362730 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0b57ffe3-7afa-421f-b8f2-330b43a71353" containerName="revoke-keys" Oct 29 22:37:22.362744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362739 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b57ffe3-7afa-421f-b8f2-330b43a71353" containerName="revoke-keys" Oct 29 22:37:22.362849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362795 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7c9579b0-9d8f-4caa-babc-69ba5a02a012" containerName="payload-pre-processing" Oct 29 22:37:22.362849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362801 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9579b0-9d8f-4caa-babc-69ba5a02a012" containerName="payload-pre-processing" Oct 29 22:37:22.362938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362928 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9579b0-9d8f-4caa-babc-69ba5a02a012" containerName="payload-pre-processing" Oct 29 22:37:22.362969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362944 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b37d2d-9fb0-4ff3-9ce4-f4d64a21c7cb" containerName="payload-processing" Oct 29 22:37:22.362969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.362951 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b57ffe3-7afa-421f-b8f2-330b43a71353" containerName="revoke-keys" Oct 29 22:37:22.367442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.367422 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.404387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.404348 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf"] Oct 29 22:37:22.408432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.408402 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-84304f-openshift-default-dockercfg-p57fc\"" Oct 29 22:37:22.484227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484197 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484234 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484340 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484389 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484482 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484538 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484623 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484690 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dctw\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.484820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.484806 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.607773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.607100 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.607773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.607350 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.608049 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.609050 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.609309 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.609585 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.609186 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.609663 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.609862 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dctw\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.610444 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.610996 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.615081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.613642 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.618775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.617568 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.618775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.617840 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.622553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.622519 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.625777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.625750 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.629416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.629384 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.636066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.636045 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dctw\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:22.686161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:22.686121 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:23.092484 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:37:23.091958 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e462c93_e10b_4572_b36f_b934c912f870.slice/crio-e0f57bcf2a62c79b11ba1c0e850297a707852c647f0c3ee0a1da42130a8a6f0e WatchSource:0}: Error finding container e0f57bcf2a62c79b11ba1c0e850297a707852c647f0c3ee0a1da42130a8a6f0e: Status 404 returned error can't find the container with id e0f57bcf2a62c79b11ba1c0e850297a707852c647f0c3ee0a1da42130a8a6f0e Oct 29 22:37:23.098362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.098318 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:23.098628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.098605 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf"] Oct 29 22:37:23.098813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.098778 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:23.098964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.098949 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:23.702627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.699745 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x"] Oct 29 22:37:23.709019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.708964 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.711350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.711302 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x"] Oct 29 22:37:23.729128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.725380 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d478b36e-openshift-default-dockercfg-6hzxg\"" Oct 29 22:37:23.798050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.797947 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.797990 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zjh\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798124 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798165 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798331 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798368 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798399 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798431 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.798688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.798498 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900157 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900295 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900314 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zjh\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900393 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900424 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900596 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900645 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900678 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900722 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.900890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.900805 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.901165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.901139 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.901281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.901192 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.901281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.901225 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.901402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.901282 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.902944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.902894 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.903304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.903286 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.922231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.922191 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:23.924308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:23.924279 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zjh\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:24.018189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.018098 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" event={"ID":"1e462c93-e10b-4572-b36f-b934c912f870","Type":"ContainerStarted","Data":"484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136"} Oct 29 22:37:24.018189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.018155 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" event={"ID":"1e462c93-e10b-4572-b36f-b934c912f870","Type":"ContainerStarted","Data":"e0f57bcf2a62c79b11ba1c0e850297a707852c647f0c3ee0a1da42130a8a6f0e"} Oct 29 22:37:24.048792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.048748 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:24.704461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.704428 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:24.723900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.723868 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:24.744511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.744444 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podStartSLOduration=2.744428438 podStartE2EDuration="2.744428438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:24.080066275 +0000 UTC m=+1310.974141196" watchObservedRunningTime="2026-10-29 22:37:24.744428438 +0000 UTC m=+1311.638503358" Oct 29 22:37:24.745537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.745512 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x"] Oct 29 22:37:24.750394 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:37:24.750369 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2667a74a_dee8_425f_87be_3d9a39bba51e.slice/crio-6fcb9a6b7c7445aebbbdd9350fb9e6a9c666845696b88c9b85789211e9dabe9e WatchSource:0}: Error finding container 6fcb9a6b7c7445aebbbdd9350fb9e6a9c666845696b88c9b85789211e9dabe9e: Status 404 returned error can't find the container with id 6fcb9a6b7c7445aebbbdd9350fb9e6a9c666845696b88c9b85789211e9dabe9e Oct 29 22:37:24.772724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.753905 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:24.772724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.766879 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:24.772724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:24.766937 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:25.037620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.037442 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" event={"ID":"2667a74a-dee8-425f-87be-3d9a39bba51e","Type":"ContainerStarted","Data":"9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2"} Oct 29 22:37:25.037620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.037508 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:25.037620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.037533 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" event={"ID":"2667a74a-dee8-425f-87be-3d9a39bba51e","Type":"ContainerStarted","Data":"6fcb9a6b7c7445aebbbdd9350fb9e6a9c666845696b88c9b85789211e9dabe9e"} Oct 29 22:37:25.038931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.038883 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:37:25.043113 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.043092 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd"] Oct 29 22:37:25.049314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.049291 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:25.049427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.049340 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aedb82-5c6zd"] Oct 29 22:37:25.051278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.051255 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:37:25.051380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.051314 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:37:25.146104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.146040 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podStartSLOduration=2.146020416 podStartE2EDuration="2.146020416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:25.096683912 +0000 UTC m=+1311.990758834" watchObservedRunningTime="2026-10-29 22:37:25.146020416 +0000 UTC m=+1312.040095340" Oct 29 22:37:25.230814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:25.230784 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad0113a-2752-4a4a-92a3-8eca635a4f9f" path="/var/lib/kubelet/pods/8ad0113a-2752-4a4a-92a3-8eca635a4f9f/volumes" Oct 29 22:37:26.069723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:26.053860 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:27.079160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:27.079125 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:27.080378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:27.080352 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:37:28.447568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.447536 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb"] Oct 29 22:37:28.453584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.453562 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.482527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.482476 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb"] Oct 29 22:37:28.486924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.486896 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-84304f\"" Oct 29 22:37:28.536039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.528476 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dx6\" (UniqueName: \"kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6\") pod \"payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.536039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.528566 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.562723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.555494 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-84304f-dockercfg-294vt\"" Oct 29 22:37:28.656722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.650181 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dx6\" (UniqueName: \"kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6\") pod \"payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.656722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.650272 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.656722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.651419 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.706926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.705694 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dx6\" (UniqueName: \"kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6\") pod \"payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:28.791488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:28.791454 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:29.148133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:29.148100 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb"] Oct 29 22:37:30.107624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.107588 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" event={"ID":"047b35a0-9d90-4bc0-a00c-95cdc2c191e2","Type":"ContainerStarted","Data":"3cb638d99d8b13bba67138dc62a3bc69285b5503b5a84e46836a0bbf5dc2f17c"} Oct 29 22:37:30.108061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.107658 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:30.108061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.107692 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" event={"ID":"047b35a0-9d90-4bc0-a00c-95cdc2c191e2","Type":"ContainerStarted","Data":"8ee05ddda84f9373382dd520e2acb8e097caa833136ed6284e3b24c7a3b07f4b"} Oct 29 22:37:30.148677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.148616 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" podStartSLOduration=2.148599747 podStartE2EDuration="2.148599747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:30.147373371 +0000 UTC m=+1317.041448293" watchObservedRunningTime="2026-10-29 22:37:30.148599747 +0000 UTC m=+1317.042674669" Oct 29 22:37:30.167217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.166883 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:37:30.167217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.166944 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:37:30.174665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.174629 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg"] Oct 29 22:37:30.182340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.179505 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.195775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.195051 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d478b36e\"" Oct 29 22:37:30.195775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.195296 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d478b36e-dockercfg-4z7v2\"" Oct 29 22:37:30.200947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.200920 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg"] Oct 29 22:37:30.284721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.282333 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.284721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.282616 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjmt\" (UniqueName: \"kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.384464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.384365 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjmt\" (UniqueName: \"kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.384654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.384578 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.386281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.385517 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.421763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.417499 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjmt\" (UniqueName: \"kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.505719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.498925 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:30.886041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:30.886012 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg"] Oct 29 22:37:31.038483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.038430 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c"] Oct 29 22:37:31.044068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.044035 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d21112-6ds5c"] Oct 29 22:37:31.119106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.119066 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" event={"ID":"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0","Type":"ContainerStarted","Data":"232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4"} Oct 29 22:37:31.119599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.119133 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:31.119599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.119153 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" event={"ID":"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0","Type":"ContainerStarted","Data":"6711b7b35041a639ff729e3ea0f9c47b120827771ca6bbda1b29a73ab768ff06"} Oct 29 22:37:31.165563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.165443 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" podStartSLOduration=1.165429176 podStartE2EDuration="1.165429176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:31.159323432 +0000 UTC m=+1318.053398354" watchObservedRunningTime="2026-10-29 22:37:31.165429176 +0000 UTC m=+1318.059504095" Oct 29 22:37:31.224292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:31.224259 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13924bfc-cde2-4e61-ba68-539f640ad853" path="/var/lib/kubelet/pods/13924bfc-cde2-4e61-ba68-539f640ad853/volumes" Oct 29 22:37:33.067570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.067540 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9"] Oct 29 22:37:33.083462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.083432 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1618b90-sjnf9"] Oct 29 22:37:33.141143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.141112 5367 generic.go:356] "Generic (PLEG): container finished" podID="4887aad4-4623-463e-a66c-f470439dc173" containerID="9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4" exitCode=0 Oct 29 22:37:33.141317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.141191 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" event={"ID":"4887aad4-4623-463e-a66c-f470439dc173","Type":"ContainerDied","Data":"9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4"} Oct 29 22:37:33.141779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.141759 5367 scope.go:122] "RemoveContainer" containerID="9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4" Oct 29 22:37:33.143505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.143473 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:33.143599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.143558 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:33.143673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.143610 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:33.234331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:33.234298 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d84bbb7-c2f8-49b9-9116-5adb01f9db12" path="/var/lib/kubelet/pods/7d84bbb7-c2f8-49b9-9116-5adb01f9db12/volumes" Oct 29 22:37:34.153270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.153239 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" event={"ID":"4887aad4-4623-463e-a66c-f470439dc173","Type":"ContainerStarted","Data":"957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a"} Oct 29 22:37:34.248549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.248522 5367 scope.go:122] "RemoveContainer" containerID="226e4ab0bfcd9c59ba396c58ce68f31124e118ec39670dedc6292ef60b97b3fe" Oct 29 22:37:34.276835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.271106 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:37:34.276835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.271249 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:37:34.276835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.271296 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 29 22:37:34.285279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.285254 5367 scope.go:122] "RemoveContainer" containerID="b0fa4ed0cdf4fe4fdc06212da4921b8adb501fb243e5b5ca48c08a1c08630f74" Oct 29 22:37:34.297313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.297284 5367 scope.go:122] "RemoveContainer" containerID="50e46f0231c3f258e31d5073868ddcba61ae7e5a5261a4b40740e051546c50bd" Oct 29 22:37:34.308320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.308297 5367 scope.go:122] "RemoveContainer" containerID="3a61a110c56a2c94b8d881a184c54c504e61666d33788f7257b5d300c8dbaa82" Oct 29 22:37:34.318307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.318290 5367 scope.go:122] "RemoveContainer" containerID="30ee3a7973bc3560939a146fb449430c998d77f6322f326102a2aa6ba7a9e84f" Oct 29 22:37:34.330573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:34.330550 5367 scope.go:122] "RemoveContainer" containerID="192a8ecc94ba5de2b953eacb5d07cf1a021b238bd61d3e2e909008a8f9e182da" Oct 29 22:37:35.277721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:35.277121 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:37:35.277721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:35.277182 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 29 22:37:36.275723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:36.273071 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:37:36.275723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:36.273136 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 29 22:37:37.275722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:37.275138 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:37:37.275722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:37.275204 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 29 22:37:38.278721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:38.275085 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:37:38.278721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:38.275153 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 29 22:37:39.326647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:39.326613 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:37:39.327216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:39.327189 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:37:39.328412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:39.328393 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 22:37:41.122342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:41.122312 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 22:37:42.133538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:42.133511 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:37:47.032106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:47.032070 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j"] Oct 29 22:37:47.044578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:47.044550 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85fccdc-khq8j"] Oct 29 22:37:47.224658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:47.224626 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e7a230-31aa-4b8d-80e5-78f9d0b1627d" path="/var/lib/kubelet/pods/d7e7a230-31aa-4b8d-80e5-78f9d0b1627d/volumes" Oct 29 22:37:48.541173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.541141 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h"] Oct 29 22:37:48.546378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.546352 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.570146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.570114 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h"] Oct 29 22:37:48.617396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.617361 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0ab33e-openshift-default-dockercfg-gm29v\"" Oct 29 22:37:48.658594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.658559 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.658811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.658626 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.658811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.658777 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.658911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.658817 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.658957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.658921 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.658957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.658949 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.659063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.659050 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.659117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.659105 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.659177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.659165 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7qw\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761077 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761159 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761195 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761259 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761287 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761412 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761462 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761548 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7qw\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761570 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761654 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761771 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761797 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.761978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.761855 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.762195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.762178 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.763572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.763552 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.763919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.763901 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.788053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.788031 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7qw\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.788409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.788386 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:48.860995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:48.860962 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:49.269610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:49.267437 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h"] Oct 29 22:37:49.280843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:49.280066 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:49.280843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:49.280183 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:49.280843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:49.280224 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:49.348272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:49.348234 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" event={"ID":"c7c362d6-ca73-4287-96e3-08b67ca51383","Type":"ContainerStarted","Data":"e737af3de37cda77b2bb8e526d699c271993226cd073765dcdcd88afccb6853a"} Oct 29 22:37:50.360603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:50.360566 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" event={"ID":"c7c362d6-ca73-4287-96e3-08b67ca51383","Type":"ContainerStarted","Data":"79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88"} Oct 29 22:37:50.440514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:50.440459 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podStartSLOduration=2.440443529 podStartE2EDuration="2.440443529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:50.421988675 +0000 UTC m=+1337.316063594" watchObservedRunningTime="2026-10-29 22:37:50.440443529 +0000 UTC m=+1337.334518449" Oct 29 22:37:50.861786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:50.861751 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:50.869881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:50.869854 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:51.037827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:51.037796 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6"] Oct 29 22:37:51.044029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:51.044001 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191ddbbe-pfjt6"] Oct 29 22:37:51.225635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:51.225551 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ceac8e-6bd2-4a5d-92b6-7a2fb900df27" path="/var/lib/kubelet/pods/49ceac8e-6bd2-4a5d-92b6-7a2fb900df27/volumes" Oct 29 22:37:51.443007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:51.432207 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:51.443007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:51.438182 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:37:52.336071 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:37:52.336036 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:37:53.102171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:53.102126 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm"] Oct 29 22:37:53.116422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:53.116386 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412b45ba-blffm"] Oct 29 22:37:53.226501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:53.226464 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c7762e-2cae-4ae4-8eb1-35368d66f02b" path="/var/lib/kubelet/pods/83c7762e-2cae-4ae4-8eb1-35368d66f02b/volumes" Oct 29 22:37:54.980723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:54.975627 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9"] Oct 29 22:37:55.164211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.145030 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.167347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.167318 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0ab33e-dockercfg-sggz8\"" Oct 29 22:37:55.181892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.180805 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0ab33e\"" Oct 29 22:37:55.292259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.292165 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.298623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.298589 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7n6j\" (UniqueName: \"kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j\") pod \"payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.345125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.345093 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9"] Oct 29 22:37:55.427571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.427526 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.427811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.427645 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7n6j\" (UniqueName: \"kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j\") pod \"payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.428366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.428336 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.449403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.449373 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7n6j\" (UniqueName: \"kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j\") pod \"payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.549355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.549263 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:55.943084 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:37:55.943039 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c191ad_46e4_451e_afe2_ce43bead4d30.slice/crio-9b8b4c45b676c6984f797772c328b0837d114421d3e39cae47ab47bac1f20123 WatchSource:0}: Error finding container 9b8b4c45b676c6984f797772c328b0837d114421d3e39cae47ab47bac1f20123: Status 404 returned error can't find the container with id 9b8b4c45b676c6984f797772c328b0837d114421d3e39cae47ab47bac1f20123 Oct 29 22:37:55.944219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:55.944194 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9"] Oct 29 22:37:56.589593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:56.589548 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" event={"ID":"e2c191ad-46e4-451e-afe2-ce43bead4d30","Type":"ContainerStarted","Data":"7b16186dc7a7a4946e654d94ddd8458260de1f20da0e3a8dc232b705951272f2"} Oct 29 22:37:56.590129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:56.589623 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" event={"ID":"e2c191ad-46e4-451e-afe2-ce43bead4d30","Type":"ContainerStarted","Data":"9b8b4c45b676c6984f797772c328b0837d114421d3e39cae47ab47bac1f20123"} Oct 29 22:37:56.590129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:56.589656 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:37:56.633518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:56.633386 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" podStartSLOduration=2.633365745 podStartE2EDuration="2.633365745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:56.629572698 +0000 UTC m=+1343.523647619" watchObservedRunningTime="2026-10-29 22:37:56.633365745 +0000 UTC m=+1343.527440667" Oct 29 22:37:58.150292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.150259 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb"] Oct 29 22:37:58.160723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.154357 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.182064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.181206 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb"] Oct 29 22:37:58.186730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.186000 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-6bb06a\"" Oct 29 22:37:58.342795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.342735 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.343156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.343136 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qlb\" (UniqueName: \"kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb\") pod \"payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.444628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.444532 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qlb\" (UniqueName: \"kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb\") pod \"payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.444826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.444793 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.445540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.445517 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.478576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.478541 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qlb\" (UniqueName: \"kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb\") pod \"payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.521309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:58.520672 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:58.993928 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:37:58.993889 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade12afa_0df7_4590_858b_8e6bc8a9977a.slice/crio-05e5952da19f75d999e6a793171d262b534c8640f995ceaae0a47690fa42c012 WatchSource:0}: Error finding container 05e5952da19f75d999e6a793171d262b534c8640f995ceaae0a47690fa42c012: Status 404 returned error can't find the container with id 05e5952da19f75d999e6a793171d262b534c8640f995ceaae0a47690fa42c012 Oct 29 22:37:59.005924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:59.005887 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb"] Oct 29 22:37:59.692403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:59.692363 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" event={"ID":"ade12afa-0df7-4590-858b-8e6bc8a9977a","Type":"ContainerStarted","Data":"bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097"} Oct 29 22:37:59.692823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:59.692427 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" event={"ID":"ade12afa-0df7-4590-858b-8e6bc8a9977a","Type":"ContainerStarted","Data":"05e5952da19f75d999e6a793171d262b534c8640f995ceaae0a47690fa42c012"} Oct 29 22:37:59.692823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:59.692512 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:37:59.739549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:37:59.739503 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" podStartSLOduration=1.739489281 podStartE2EDuration="1.739489281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:59.734211689 +0000 UTC m=+1346.628286614" watchObservedRunningTime="2026-10-29 22:37:59.739489281 +0000 UTC m=+1346.633564202" Oct 29 22:38:00.656303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.656271 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd"] Oct 29 22:38:00.660248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.660230 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.676603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.676574 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f76fb5\"" Oct 29 22:38:00.682832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.682809 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd"] Oct 29 22:38:00.688952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.688917 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f76fb5-dockercfg-j2hq5\"" Oct 29 22:38:00.718526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.718493 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728"] Oct 29 22:38:00.722459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.722391 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.740780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.740751 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ssm\" (UniqueName: \"kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm\") pod \"payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.744528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.744498 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.769819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.761325 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728"] Oct 29 22:38:00.854318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.847057 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.854318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.847237 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ssm\" (UniqueName: \"kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm\") pod \"payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.854318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.847360 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.854318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.847524 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmwc\" (UniqueName: \"kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc\") pod \"payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.854318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.849008 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.949280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.901201 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ssm\" (UniqueName: \"kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm\") pod \"payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:00.949624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.949440 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmwc\" (UniqueName: \"kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc\") pod \"payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.949828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.949743 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.950543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.950516 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.970866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.970805 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmwc\" (UniqueName: \"kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc\") pod \"payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:00.977013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:00.975996 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:01.051304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.051055 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:01.345724 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:38:01.345580 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d70bc4f_1677_4043_9da2_fe872c4a92e6.slice/crio-17649a96140fbc2f581a91f6682d5c769041534041e5e1ad1499103c80f70a52 WatchSource:0}: Error finding container 17649a96140fbc2f581a91f6682d5c769041534041e5e1ad1499103c80f70a52: Status 404 returned error can't find the container with id 17649a96140fbc2f581a91f6682d5c769041534041e5e1ad1499103c80f70a52 Oct 29 22:38:01.349402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.349374 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd"] Oct 29 22:38:01.416210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.416177 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728"] Oct 29 22:38:01.416412 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:38:01.416382 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf147bf4c_527f_4cf7_85b1_58fa6b2eedb8.slice/crio-d10861d696697d5aa41555d6f89c1a8b40b0eb6fe1e0ae76cf5f821f3662cbfc WatchSource:0}: Error finding container d10861d696697d5aa41555d6f89c1a8b40b0eb6fe1e0ae76cf5f821f3662cbfc: Status 404 returned error can't find the container with id d10861d696697d5aa41555d6f89c1a8b40b0eb6fe1e0ae76cf5f821f3662cbfc Oct 29 22:38:01.717923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.717887 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" event={"ID":"5d70bc4f-1677-4043-9da2-fe872c4a92e6","Type":"ContainerStarted","Data":"b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d"} Oct 29 22:38:01.718123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.717953 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" event={"ID":"5d70bc4f-1677-4043-9da2-fe872c4a92e6","Type":"ContainerStarted","Data":"17649a96140fbc2f581a91f6682d5c769041534041e5e1ad1499103c80f70a52"} Oct 29 22:38:01.718123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.717986 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:01.720019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.719994 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" event={"ID":"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8","Type":"ContainerStarted","Data":"22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948"} Oct 29 22:38:01.720359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.720037 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:01.720359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.720052 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" event={"ID":"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8","Type":"ContainerStarted","Data":"d10861d696697d5aa41555d6f89c1a8b40b0eb6fe1e0ae76cf5f821f3662cbfc"} Oct 29 22:38:01.759211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.759092 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" podStartSLOduration=1.7590774059999998 podStartE2EDuration="1.759077406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:01.756967696 +0000 UTC m=+1348.651042618" watchObservedRunningTime="2026-10-29 22:38:01.759077406 +0000 UTC m=+1348.653152326" Oct 29 22:38:01.800018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:01.799966 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" podStartSLOduration=1.7999509759999999 podStartE2EDuration="1.799950976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:01.795802403 +0000 UTC m=+1348.689877326" watchObservedRunningTime="2026-10-29 22:38:01.799950976 +0000 UTC m=+1348.694025897" Oct 29 22:38:05.767913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:05.767881 5367 generic.go:356] "Generic (PLEG): container finished" podID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerID="b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71" exitCode=0 Oct 29 22:38:05.768350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:05.767954 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" event={"ID":"2e312dd8-3618-4b93-9fc6-321f2460aa77","Type":"ContainerDied","Data":"b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71"} Oct 29 22:38:05.768461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:05.768445 5367 scope.go:122] "RemoveContainer" containerID="b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71" Oct 29 22:38:05.770365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:05.770328 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:05.770499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:05.770425 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:05.770499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:05.770473 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:06.030382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:06.030295 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr"] Oct 29 22:38:06.035639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:06.035608 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cef486-nc2rr"] Oct 29 22:38:06.793958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:06.793925 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" event={"ID":"2e312dd8-3618-4b93-9fc6-321f2460aa77","Type":"ContainerStarted","Data":"13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6"} Oct 29 22:38:07.224162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:07.224127 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50350920-7b34-49ab-871a-0dca19d59fab" path="/var/lib/kubelet/pods/50350920-7b34-49ab-871a-0dca19d59fab/volumes" Oct 29 22:38:07.603618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:07.603590 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:38:07.691779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:07.691748 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:38:07.692915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:07.692888 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:38:07.693048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:07.692944 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 29 22:38:08.696127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:08.696086 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:38:08.696763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:08.696730 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 29 22:38:09.030607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:09.030528 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7"] Oct 29 22:38:09.043621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:09.043593 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17769705-ssmr7"] Oct 29 22:38:09.239509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:09.239465 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f8756e-29f9-405f-9881-fa3c62fdbdee" path="/var/lib/kubelet/pods/00f8756e-29f9-405f-9881-fa3c62fdbdee/volumes" Oct 29 22:38:09.692376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:09.692339 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:38:09.692650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:09.692400 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 29 22:38:10.691925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:10.691880 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:38:10.692430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:10.691939 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 29 22:38:10.710999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:10.710970 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 22:38:11.576724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:11.575817 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:38:11.692664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:11.692626 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:38:11.693142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:11.692683 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 29 22:38:12.697060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:12.696984 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:38:12.698202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:12.698181 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 22:38:12.735736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:12.735671 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:38:12.735904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:12.735762 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:38:19.084441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:19.084349 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5"] Oct 29 22:38:19.090916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:19.090880 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2df52cd-bb5m5"] Oct 29 22:38:19.225961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:19.225916 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf52d625-9d3b-4681-bb1c-8c827aaf2335" path="/var/lib/kubelet/pods/bf52d625-9d3b-4681-bb1c-8c827aaf2335/volumes" Oct 29 22:38:21.074795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:21.074753 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt"] Oct 29 22:38:21.086723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:21.082786 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0d7f6-gw2wt"] Oct 29 22:38:21.315427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:21.315384 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290d4e6c-5515-41b6-a409-c205cbe20432" path="/var/lib/kubelet/pods/290d4e6c-5515-41b6-a409-c205cbe20432/volumes" Oct 29 22:38:22.336497 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:38:22.336466 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:38:24.676170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.676127 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm"] Oct 29 22:38:24.684607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.683618 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.717888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.714914 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a26426-openshift-default-dockercfg-dpvzx\"" Oct 29 22:38:24.718242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.718147 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm"] Oct 29 22:38:24.881175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.881138 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.881641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.881599 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.882029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.881997 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.882256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.882239 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.882577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.882560 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.882891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.882856 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.883090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.883057 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.883374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.883319 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:24.883529 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:24.883512 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fws55\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.004092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.004015 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.004374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.004359 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.004820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.004805 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.005006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.004901 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.005090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.004757 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.005166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.005139 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.005521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.005274 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.006025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.005974 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.006152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.006136 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.006287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.006273 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.006807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.006788 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.007369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.007350 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fws55\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.007843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.007825 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.008342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.008229 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.030720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.012213 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.030720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.012422 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.046033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.045627 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fws55\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.082789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.080454 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token\") pod \"e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.324450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.321106 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:25.692343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.692301 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm"] Oct 29 22:38:25.692885 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:38:25.692850 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722caee6_f913_4d87_a3b3_9342ccc42ebe.slice/crio-9705aab95c12e646a8a371ce946a42cf91f1d83b0984d9eedb4bda7ed99e20c3 WatchSource:0}: Error finding container 9705aab95c12e646a8a371ce946a42cf91f1d83b0984d9eedb4bda7ed99e20c3: Status 404 returned error can't find the container with id 9705aab95c12e646a8a371ce946a42cf91f1d83b0984d9eedb4bda7ed99e20c3 Oct 29 22:38:25.696517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.696482 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:25.696622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.696553 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:25.696622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:25.696581 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:26.083722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.082800 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:38:26.083722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.082871 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:26.170885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.169481 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" event={"ID":"722caee6-f913-4d87-a3b3-9342ccc42ebe","Type":"ContainerStarted","Data":"4548dd87f251184c4db6b87237a44e023d49ea243c794942a2109c91512bb3c5"} Oct 29 22:38:26.170885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.169553 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" event={"ID":"722caee6-f913-4d87-a3b3-9342ccc42ebe","Type":"ContainerStarted","Data":"9705aab95c12e646a8a371ce946a42cf91f1d83b0984d9eedb4bda7ed99e20c3"} Oct 29 22:38:26.220770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.220691 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podStartSLOduration=2.22066917 podStartE2EDuration="2.22066917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:26.21427191 +0000 UTC m=+1373.108346832" watchObservedRunningTime="2026-10-29 22:38:26.22066917 +0000 UTC m=+1373.114744092" Oct 29 22:38:26.331878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.331798 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:26.337019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.336588 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:38:26.337019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.336652 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:38:26.569245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.569211 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x"] Oct 29 22:38:26.577569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.577543 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.611521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.611482 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x"] Oct 29 22:38:26.652274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.652195 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgmv\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.652476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.652356 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.655251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.653419 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.655251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.653483 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.655251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.653512 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.655251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.653542 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.655251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.653635 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.755640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.755604 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgmv\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.756163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.755762 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.756163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.755867 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.756163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.755900 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.756163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.755938 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.756163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.755979 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.756163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.756096 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.757403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.757351 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.759205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.758987 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.759825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.759782 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.760757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.760694 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.760989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.760965 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.776837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.776801 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgmv\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.777408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.777385 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6q2x\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:26.894807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:26.894685 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:27.270391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:27.270275 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x"] Oct 29 22:38:27.287046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:27.287009 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:27.287238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:27.287215 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:27.329129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:27.329095 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:28.137401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.124453 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:38:28.137401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.124525 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:28.288028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.286859 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" event={"ID":"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa","Type":"ContainerStarted","Data":"aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef"} Oct 29 22:38:28.288028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.286933 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:28.288028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.286958 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" event={"ID":"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa","Type":"ContainerStarted","Data":"3f0777ce13073bad733e886e6e64f855fe2dc43bb7445658f04808f48a6d013a"} Oct 29 22:38:28.288446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.288419 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:28.290024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.289986 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:38:28.293809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.293737 5367 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-z6q2x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:38:28.294068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.294016 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" podUID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:38:28.363129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:28.361989 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" podStartSLOduration=2.361968413 podStartE2EDuration="2.361968413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:28.338803836 +0000 UTC m=+1375.232878766" watchObservedRunningTime="2026-10-29 22:38:28.361968413 +0000 UTC m=+1375.256043340" Oct 29 22:38:29.296789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:29.296746 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:38:31.335745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.335712 5367 generic.go:356] "Generic (PLEG): container finished" podID="1e462c93-e10b-4572-b36f-b934c912f870" containerID="484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136" exitCode=0 Oct 29 22:38:31.336262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.335773 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" event={"ID":"1e462c93-e10b-4572-b36f-b934c912f870","Type":"ContainerDied","Data":"484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136"} Oct 29 22:38:31.336337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.336314 5367 scope.go:122] "RemoveContainer" containerID="484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136" Oct 29 22:38:31.337853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.337830 5367 generic.go:356] "Generic (PLEG): container finished" podID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerID="9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2" exitCode=0 Oct 29 22:38:31.337939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.337907 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" event={"ID":"2667a74a-dee8-425f-87be-3d9a39bba51e","Type":"ContainerDied","Data":"9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2"} Oct 29 22:38:31.338363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.338338 5367 scope.go:122] "RemoveContainer" containerID="9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2" Oct 29 22:38:31.338470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.338442 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:31.338545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.338528 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:31.338594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.338579 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:31.339990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.339957 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:31.340097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.340042 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:31.340159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:31.340109 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:38:32.045740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.045680 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg"] Oct 29 22:38:32.055401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.055158 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c373c4-tc9qg"] Oct 29 22:38:32.366627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.365641 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" event={"ID":"1e462c93-e10b-4572-b36f-b934c912f870","Type":"ContainerStarted","Data":"660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85"} Oct 29 22:38:32.373742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.370813 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" event={"ID":"2667a74a-dee8-425f-87be-3d9a39bba51e","Type":"ContainerStarted","Data":"000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108"} Oct 29 22:38:32.687330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.687156 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:38:32.716749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.689741 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 29 22:38:32.716749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:32.689817 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 29 22:38:33.042459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.042370 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr"] Oct 29 22:38:33.049024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.048986 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:38:33.049232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.049206 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:38:33.049311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.049272 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:38:33.110723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.084439 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad3e8be-p5lsr"] Oct 29 22:38:33.227116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.227078 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b077c7c9-cbe7-47d0-a12e-e045abfe2158" path="/var/lib/kubelet/pods/b077c7c9-cbe7-47d0-a12e-e045abfe2158/volumes" Oct 29 22:38:33.227593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.227574 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e2d128-35b4-48b4-b53d-f9a6085556b7" path="/var/lib/kubelet/pods/f5e2d128-35b4-48b4-b53d-f9a6085556b7/volumes" Oct 29 22:38:33.235560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.235538 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4"] Oct 29 22:38:33.240188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.240167 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.288748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.270685 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a26426-dockercfg-9dr2d\"" Oct 29 22:38:33.288748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.270806 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a26426\"" Oct 29 22:38:33.369756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.368976 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.369756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.369175 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ct7b\" (UniqueName: \"kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b\") pod \"payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.391761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.372368 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4"] Oct 29 22:38:33.497752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.476048 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.497752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.476216 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ct7b\" (UniqueName: \"kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b\") pod \"payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.608750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.583308 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.695633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.695504 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 29 22:38:33.695633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.695566 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 29 22:38:33.918434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.916304 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ct7b\" (UniqueName: \"kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b\") pod \"payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.918434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.916574 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:33.926746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.926720 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn"] Oct 29 22:38:33.941053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.936022 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:33.941053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:33.939356 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn"] Oct 29 22:38:34.082734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.082247 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:38:34.082734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.082306 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:38:34.129782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.129743 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbcp\" (UniqueName: \"kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp\") pod \"payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.129979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.129955 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.231558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.231389 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbcp\" (UniqueName: \"kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp\") pod \"payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.231558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.231525 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.232412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.232386 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.252286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.252253 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbcp\" (UniqueName: \"kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp\") pod \"payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.426726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.425065 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:34.456570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.456534 5367 scope.go:122] "RemoveContainer" containerID="df5fa6a383eb4cc1eaefdf6e4bb4a7b93ca9c52fe7f4a601532758991eeb2f7b" Oct 29 22:38:34.456986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.456600 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4"] Oct 29 22:38:34.705777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.690909 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 29 22:38:34.705777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.690970 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 29 22:38:34.844618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:34.844589 5367 scope.go:122] "RemoveContainer" containerID="50bfbfc8bcb0db32c61fce5f5d6ef5ad1da449dff7721c3042b2283d66889dab" Oct 29 22:38:35.054213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.054166 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:38:35.054493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.054449 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:38:35.057551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.057468 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" event={"ID":"358770a0-e2ae-42ef-a765-a96e610de1b0","Type":"ContainerStarted","Data":"86e702d0a43e12fa09e342404d2ae08e32dfc4b8d46fb366398e2cf9a0aaf117"} Oct 29 22:38:35.196725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.144772 5367 scope.go:122] "RemoveContainer" containerID="5ba1a313a4405ff324bb885a95614138d35847e19569e103aea7612a5916d164" Oct 29 22:38:35.336485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.336458 5367 scope.go:122] "RemoveContainer" containerID="292f2736ae873edf9082be48935e844cefc86f1f7620dfd0a7999a1f49ebe7b0" Oct 29 22:38:35.350345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.349924 5367 scope.go:122] "RemoveContainer" containerID="546524d927559c3e4e606bb59352ce166e839900dc1de497c3b282ee1dfd64de" Oct 29 22:38:35.427323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.427289 5367 scope.go:122] "RemoveContainer" containerID="7303089e2f05b77b183812191ebf5dd4c437dbcd0e30166bf8a120258913ee1e" Oct 29 22:38:35.441210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.441186 5367 scope.go:122] "RemoveContainer" containerID="0ad892a3a09c4cc3c893537d1adb29c08381b84af7249334f84c436c12ec85de" Oct 29 22:38:35.515338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.515310 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn"] Oct 29 22:38:35.517514 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:38:35.517487 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6c03f3_a7ad_4e26_9333_2668607d1ff1.slice/crio-a2cf5b04602d1556ccbcab13ffabac10afc48efc8047fe3287580ef7fb34b329 WatchSource:0}: Error finding container a2cf5b04602d1556ccbcab13ffabac10afc48efc8047fe3287580ef7fb34b329: Status 404 returned error can't find the container with id a2cf5b04602d1556ccbcab13ffabac10afc48efc8047fe3287580ef7fb34b329 Oct 29 22:38:35.687482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.687050 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 29 22:38:35.687482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:35.687109 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 29 22:38:36.050104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.050024 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:38:36.050104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.050077 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:38:36.118233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.118199 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" event={"ID":"358770a0-e2ae-42ef-a765-a96e610de1b0","Type":"ContainerStarted","Data":"b372505febf57337b95d6a74986e9c60faef4ccefa94d8e35a619b6293f13c2b"} Oct 29 22:38:36.118954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.118930 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:36.121525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.121505 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" event={"ID":"8e6c03f3-a7ad-4e26-9333-2668607d1ff1","Type":"ContainerStarted","Data":"95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9"} Oct 29 22:38:36.121657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.121644 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" event={"ID":"8e6c03f3-a7ad-4e26-9333-2668607d1ff1","Type":"ContainerStarted","Data":"a2cf5b04602d1556ccbcab13ffabac10afc48efc8047fe3287580ef7fb34b329"} Oct 29 22:38:36.121788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.121777 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:36.215730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.214580 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" podStartSLOduration=3.214561727 podStartE2EDuration="3.214561727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:36.197551266 +0000 UTC m=+1383.091626190" watchObservedRunningTime="2026-10-29 22:38:36.214561727 +0000 UTC m=+1383.108636648" Oct 29 22:38:36.245013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.244970 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" podStartSLOduration=3.244956267 podStartE2EDuration="3.244956267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:36.238614682 +0000 UTC m=+1383.132689604" watchObservedRunningTime="2026-10-29 22:38:36.244956267 +0000 UTC m=+1383.139031190" Oct 29 22:38:36.699723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.696855 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 29 22:38:36.699723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:36.696933 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 29 22:38:37.049308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:37.049220 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:38:37.049308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:37.049293 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:38:37.692727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:37.692186 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 29 22:38:37.692727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:37.692289 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 29 22:38:38.050029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:38.049954 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 22:38:38.050029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:38.050006 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 22:38:38.691857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:38.691825 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:38:38.692322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:38.692302 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:38:38.693076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:38.693050 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 22:38:39.054145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:39.054122 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:38:39.054628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:39.054611 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:38:39.055337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:39.055314 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:38:47.033313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:47.033278 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6"] Oct 29 22:38:47.049790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:47.049754 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3f9ba2-tl9k6"] Oct 29 22:38:47.198957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:47.198926 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:38:47.199145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:47.199021 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:38:47.225588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:47.225558 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e545ed-6331-47a0-bff1-bd549b60f753" path="/var/lib/kubelet/pods/32e545ed-6331-47a0-bff1-bd549b60f753/volumes" Oct 29 22:38:48.033102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:48.033065 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9"] Oct 29 22:38:48.044855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:48.044825 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071309d2-5c4g9"] Oct 29 22:38:49.225230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:49.225198 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f95ec48-ba67-4903-b572-1972373c3f33" path="/var/lib/kubelet/pods/6f95ec48-ba67-4903-b572-1972373c3f33/volumes" Oct 29 22:38:50.049783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:50.049741 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx"] Oct 29 22:38:50.072752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:50.072065 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a793618-vc5mx"] Oct 29 22:38:51.226881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:51.226845 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609a5784-7d16-49e2-9db8-074e0d3ad8c0" path="/var/lib/kubelet/pods/609a5784-7d16-49e2-9db8-074e0d3ad8c0/volumes" Oct 29 22:38:52.338793 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:38:52.338752 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:38:53.029289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:53.029252 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b"] Oct 29 22:38:53.042715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:53.042676 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-433293e7-2gc7b"] Oct 29 22:38:53.233189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:53.233138 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0700e067-22a1-4234-bee0-b179b14d52ca" path="/var/lib/kubelet/pods/0700e067-22a1-4234-bee0-b179b14d52ca/volumes" Oct 29 22:38:57.047414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:57.047351 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g"] Oct 29 22:38:57.058455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:57.058427 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dfdd1c-tqs8g"] Oct 29 22:38:57.230482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:38:57.230443 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413807bc-aee3-474b-a26e-e1f9ac27e8fc" path="/var/lib/kubelet/pods/413807bc-aee3-474b-a26e-e1f9ac27e8fc/volumes" Oct 29 22:39:03.039247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:03.039209 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49"] Oct 29 22:39:03.051746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:03.051713 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb64c6-thm49"] Oct 29 22:39:03.227402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:03.227368 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f757d63e-02cf-4db6-828d-59ed1c554833" path="/var/lib/kubelet/pods/f757d63e-02cf-4db6-828d-59ed1c554833/volumes" Oct 29 22:39:04.821026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.820897 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28"] Oct 29 22:39:04.821536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.821189 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" containerID="cri-o://2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0" gracePeriod=30 Oct 29 22:39:04.846766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.845266 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt"] Oct 29 22:39:04.846766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.845534 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" containerID="cri-o://32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10" gracePeriod=30 Oct 29 22:39:04.880748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.880660 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt"] Oct 29 22:39:04.880966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.880932 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" containerID="cri-o://ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b" gracePeriod=30 Oct 29 22:39:04.888616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.888592 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j"] Oct 29 22:39:04.889067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:04.889039 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" containerID="cri-o://c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1" gracePeriod=30 Oct 29 22:39:05.047624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:05.047592 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj"] Oct 29 22:39:05.052495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:05.052470 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a85481-fcclj"] Oct 29 22:39:05.232039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:05.231931 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb07162-8117-40c0-92dd-d0b4e8ef54ec" path="/var/lib/kubelet/pods/acb07162-8117-40c0-92dd-d0b4e8ef54ec/volumes" Oct 29 22:39:07.474905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:07.474868 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:39:07.475593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:07.474925 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:08.029406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.029368 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms"] Oct 29 22:39:08.042114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.042082 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8948a6f8-f97ms"] Oct 29 22:39:08.166873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.166826 5367 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:39:08.167047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.166897 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:39:08.827971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.827946 5367 generic.go:356] "Generic (PLEG): container finished" podID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerID="79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88" exitCode=0 Oct 29 22:39:08.828347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.828021 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" event={"ID":"c7c362d6-ca73-4287-96e3-08b67ca51383","Type":"ContainerDied","Data":"79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88"} Oct 29 22:39:08.828521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.828501 5367 scope.go:122] "RemoveContainer" containerID="79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88" Oct 29 22:39:08.830455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.830421 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:08.830559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.830491 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:08.830559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:08.830525 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:09.232802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:09.232765 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc29e821-2074-4aad-8644-48c4d68a15dd" path="/var/lib/kubelet/pods/fc29e821-2074-4aad-8644-48c4d68a15dd/volumes" Oct 29 22:39:09.843977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:09.843933 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" event={"ID":"c7c362d6-ca73-4287-96e3-08b67ca51383","Type":"ContainerStarted","Data":"58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158"} Oct 29 22:39:09.868717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:09.865515 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:39:09.868998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:09.868965 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:09.869114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:09.869044 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:10.319761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.311542 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:39:10.348899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.348862 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.418249 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5"] Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419478 5367 cpu_manager.go:414] "containerMap: removing container" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419501 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419523 5367 cpu_manager.go:414] "containerMap: removing container" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419532 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419621 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419630 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419838 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419860 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" Oct 29 22:39:10.420798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.419871 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerName="istio-proxy" Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423212 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423295 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423382 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423425 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423443 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423464 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423495 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423551 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.423722 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6\") pod \"d3345ff1-2c5e-47c5-856a-2caac6909907\" (UID: \"d3345ff1-2c5e-47c5-856a-2caac6909907\") " Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.424459 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.427180 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.428324 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.428534 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.428740 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "kube-api-access-8cjk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.430822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.429124 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:10.430822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.429361 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.430822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.429609 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:39:10.431312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.431291 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5"] Oct 29 22:39:10.436790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.434294 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:39:10.436790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.435682 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.436790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.435791 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.438046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.438015 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo" pod "d3345ff1-2c5e-47c5-856a-2caac6909907" (UID: "d3345ff1-2c5e-47c5-856a-2caac6909907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:10.449322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.449296 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6"] Oct 29 22:39:10.450593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.450568 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" Oct 29 22:39:10.450593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.450593 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" Oct 29 22:39:10.450768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.450624 5367 cpu_manager.go:414] "containerMap: removing container" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" Oct 29 22:39:10.450768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.450632 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" Oct 29 22:39:10.450912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.450892 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" containerName="istio-proxy" Oct 29 22:39:10.450966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.450920 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerName="istio-proxy" Oct 29 22:39:10.489006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.488968 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.491569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.491479 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6"] Oct 29 22:39:10.506375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.506344 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-6bb06a-openshift-default-dockercfg-vhrww\"" Oct 29 22:39:10.526166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526143 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526181 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526209 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526245 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526482 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.526538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526506 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.526538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526531 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526606 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526645 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526682 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs\") pod \"aff1f6bc-160e-4ab6-815d-c344cd149a65\" (UID: \"aff1f6bc-160e-4ab6-815d-c344cd149a65\") " Oct 29 22:39:10.526843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.526832 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:10.527207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.527084 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.527207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.527171 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.527374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.527353 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.527374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.527357 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vvq\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.527519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.527482 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.527739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.527684 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.528176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.528152 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.528909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.528878 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.529063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.528920 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.529063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.528964 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.529063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.529029 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.529931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.529192 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.529931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.529211 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.529931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.529245 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.529931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.529291 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.529931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.529307 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530076 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530107 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3345ff1-2c5e-47c5-856a-2caac6909907-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530147 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3345ff1-2c5e-47c5-856a-2caac6909907-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530160 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1f6bc-160e-4ab6-815d-c344cd149a65-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530173 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530192 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530204 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3345ff1-2c5e-47c5-856a-2caac6909907-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530217 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530230 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjk6\" (UniqueName: \"kubernetes.io/projected/d3345ff1-2c5e-47c5-856a-2caac6909907-kube-api-access-8cjk6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.530861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530839 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "kube-api-access-7m86c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.531054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.530913 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.531188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.531061 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:10.531309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.531074 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy" pod "aff1f6bc-160e-4ab6-815d-c344cd149a65" (UID: "aff1f6bc-160e-4ab6-815d-c344cd149a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.627981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.627944 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br"] Oct 29 22:39:10.631680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.631644 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.631985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.631963 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.632140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.632126 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.632269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.632255 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.632505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.632487 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.632668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.632650 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.632814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.632797 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.633021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633005 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.633139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633124 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.633257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633241 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.633381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633367 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.633517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633504 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.633644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633630 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.633842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633828 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.633950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.633938 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.634110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.634097 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.634284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.634271 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket\") pod \"8497a33f-5655-4a84-aa30-b98c024e9645\" (UID: \"8497a33f-5655-4a84-aa30-b98c024e9645\") " Oct 29 22:39:10.634464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.634450 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert\") pod \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\" (UID: \"103bf5a5-c7de-452c-9c7f-90f83e12f0bf\") " Oct 29 22:39:10.635228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.635203 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.635555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.635384 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.635656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.635474 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:10.635751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.635585 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.636183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.636144 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br"] Oct 29 22:39:10.636299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.636278 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.636391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.636371 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.636552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.636528 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.636681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.636654 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:10.637443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.637415 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.637871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.637846 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.638103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638082 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.638263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638214 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.638360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638294 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:10.638429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638413 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638448 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638485 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638513 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638563 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638595 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638774 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.639143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.639098 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.639547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.639398 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.642453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.638690 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.642561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642483 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.642561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642538 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.642684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642579 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.642684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642664 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.642883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642745 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.642883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642861 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmkd\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.643004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642937 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.643067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.642999 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vvq\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.643192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643158 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.643276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643234 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643398 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m86c\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-kube-api-access-7m86c\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643420 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497a33f-5655-4a84-aa30-b98c024e9645-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643433 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643443 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643451 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643459 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643467 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643474 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1f6bc-160e-4ab6-815d-c344cd149a65-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643484 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643497 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643509 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643517 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643526 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497a33f-5655-4a84-aa30-b98c024e9645-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.643534 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.644511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.644074 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.649205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.647448 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "kube-api-access-2vxbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.649205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.647992 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.649205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.648114 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.670145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.669527 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "kube-api-access-zf7sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.670145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.669751 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.670145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.670096 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:10.671744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.671659 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy" pod "8497a33f-5655-4a84-aa30-b98c024e9645" (UID: "8497a33f-5655-4a84-aa30-b98c024e9645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.671863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.671827 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:10.677230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.676892 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo" pod "103bf5a5-c7de-452c-9c7f-90f83e12f0bf" (UID: "103bf5a5-c7de-452c-9c7f-90f83e12f0bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:10.701236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.700259 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.703529 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.703468 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.711385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.709615 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp"] Oct 29 22:39:10.711385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.711300 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vvq\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.714633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.714607 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.718743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.717573 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp"] Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.745810 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj678\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.745914 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstwj\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.745970 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746023 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746048 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746138 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746228 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746270 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746307 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.746353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746349 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746381 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746458 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746504 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746533 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746582 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746611 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746654 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746737 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746767 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746821 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746895 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746924 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.746990 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747031 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.747099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747069 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmkd\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747161 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747311 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747559 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxbf\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-kube-api-access-2vxbf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747584 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747660 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747676 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7sb\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-kube-api-access-zf7sb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747692 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497a33f-5655-4a84-aa30-b98c024e9645-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747729 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747742 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497a33f-5655-4a84-aa30-b98c024e9645-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.747891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.747756 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bf5a5-c7de-452c-9c7f-90f83e12f0bf-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:10.748457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.748273 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.748457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.748307 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.748457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.748334 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.749126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.749106 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:10.749126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.749111 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.749459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.749441 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.750483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.750461 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.784121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.784079 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.787387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.787364 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmkd\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.788879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.788860 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.808133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.807966 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:10.852995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.852969 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.859722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.853822 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.860478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.860248 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865098 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865172 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865212 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865268 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865296 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865345 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865369 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865526 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865641 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj678\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865739 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstwj\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865779 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865849 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865875 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.865967 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.866055 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.866099 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.866607 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.876807 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.876856 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:10.883887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.877495 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.913781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.902353 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.913781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.906953 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.915202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.907109 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.915345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.907976 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.915435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.908850 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.915506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.908911 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:10.917860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.916860 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.918713 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.918673 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.927347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.920157 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.927347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.921091 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstwj\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:10.929767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.927902 5367 generic.go:356] "Generic (PLEG): container finished" podID="aff1f6bc-160e-4ab6-815d-c344cd149a65" containerID="2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0" exitCode=0 Oct 29 22:39:10.929767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.928021 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" event={"ID":"aff1f6bc-160e-4ab6-815d-c344cd149a65","Type":"ContainerDied","Data":"2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0"} Oct 29 22:39:10.929767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.928104 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" event={"ID":"aff1f6bc-160e-4ab6-815d-c344cd149a65","Type":"ContainerDied","Data":"7b0da99035abadd77bea8a3b0f0530bd1e91ea55936fd561cb451fd204ebb1f1"} Oct 29 22:39:10.929767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.928145 5367 scope.go:122] "RemoveContainer" containerID="2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0" Oct 29 22:39:10.929767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.928368 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28" Oct 29 22:39:10.933804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.933770 5367 generic.go:356] "Generic (PLEG): container finished" podID="d3345ff1-2c5e-47c5-856a-2caac6909907" containerID="ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b" exitCode=0 Oct 29 22:39:10.934009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.933989 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" event={"ID":"d3345ff1-2c5e-47c5-856a-2caac6909907","Type":"ContainerDied","Data":"ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b"} Oct 29 22:39:10.934085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.934069 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" event={"ID":"d3345ff1-2c5e-47c5-856a-2caac6909907","Type":"ContainerDied","Data":"11a688d3df4eea86b9e92d214c76996095e8e9e994dc0d5b1613ff45f113b345"} Oct 29 22:39:10.934242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.934226 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt" Oct 29 22:39:10.944882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.944857 5367 generic.go:356] "Generic (PLEG): container finished" podID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" containerID="c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1" exitCode=0 Oct 29 22:39:10.945114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.945094 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" event={"ID":"103bf5a5-c7de-452c-9c7f-90f83e12f0bf","Type":"ContainerDied","Data":"c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1"} Oct 29 22:39:10.945262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.945246 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" event={"ID":"103bf5a5-c7de-452c-9c7f-90f83e12f0bf","Type":"ContainerDied","Data":"f746f16ea2bf4757854e7f620f2eb84ed5384a6f5c010a0b79993e3179e2757b"} Oct 29 22:39:10.945472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.945456 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j" Oct 29 22:39:10.955148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.955125 5367 generic.go:356] "Generic (PLEG): container finished" podID="8497a33f-5655-4a84-aa30-b98c024e9645" containerID="32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10" exitCode=0 Oct 29 22:39:10.956809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.956789 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" Oct 29 22:39:10.991998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.988888 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" event={"ID":"8497a33f-5655-4a84-aa30-b98c024e9645","Type":"ContainerDied","Data":"32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10"} Oct 29 22:39:10.991998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.988962 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt" event={"ID":"8497a33f-5655-4a84-aa30-b98c024e9645","Type":"ContainerDied","Data":"bba1a9f71f535f90d48ddc6477c72cdd37a8125d6c1aa0e1d8509e4b5dcd7be2"} Oct 29 22:39:11.001065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.998898 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:11.001065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.999166 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:11.001065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:10.999640 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:11.001065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.000147 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:11.008453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.008434 5367 scope.go:122] "RemoveContainer" containerID="bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648" Oct 29 22:39:11.035805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.035777 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:11.036208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.036121 5367 scope.go:122] "RemoveContainer" containerID="2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0" Oct 29 22:39:11.036583 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:11.036555 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0\": container with ID starting with 2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0 not found: ID does not exist" containerID="2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0" Oct 29 22:39:11.036673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.036594 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0"} err="failed to get container status \"2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0\": rpc error: code = NotFound desc = could not find container \"2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0\": container with ID starting with 2c42e12bdf5d858f810f0275db969d7f7f54441f73db5f91bf598f8ba64ff9a0 not found: ID does not exist" Oct 29 22:39:11.036673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.036635 5367 scope.go:122] "RemoveContainer" containerID="bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648" Oct 29 22:39:11.037253 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:11.037072 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648\": container with ID starting with bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648 not found: ID does not exist" containerID="bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648" Oct 29 22:39:11.037253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.037098 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648"} err="failed to get container status \"bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648\": rpc error: code = NotFound desc = could not find container \"bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648\": container with ID starting with bb3988f8a367098ec1277728128dbd49a9117c79a1bbab99c37c9f0a8ac55648 not found: ID does not exist" Oct 29 22:39:11.037253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.037135 5367 scope.go:122] "RemoveContainer" containerID="ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b" Oct 29 22:39:11.039255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.039230 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj678\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:11.094771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.094739 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt"] Oct 29 22:39:11.096358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.096332 5367 scope.go:122] "RemoveContainer" containerID="ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b" Oct 29 22:39:11.096926 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:11.096899 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b\": container with ID starting with ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b not found: ID does not exist" containerID="ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b" Oct 29 22:39:11.097018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.096937 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b"} err="failed to get container status \"ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b\": rpc error: code = NotFound desc = could not find container \"ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b\": container with ID starting with ddcc17907ad984637922d4311a2fd6a66065736698b4609ea2ed2d72e995c58b not found: ID does not exist" Oct 29 22:39:11.097018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.096962 5367 scope.go:122] "RemoveContainer" containerID="c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1" Oct 29 22:39:11.114486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.114404 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-w6qzt"] Oct 29 22:39:11.122160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.122110 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28"] Oct 29 22:39:11.132532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.132282 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-jjn28"] Oct 29 22:39:11.139147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.139123 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5"] Oct 29 22:39:11.198294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.198266 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j"] Oct 29 22:39:11.200171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.200146 5367 scope.go:122] "RemoveContainer" containerID="c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1" Oct 29 22:39:11.200829 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:11.200798 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1\": container with ID starting with c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1 not found: ID does not exist" containerID="c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1" Oct 29 22:39:11.200934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.200838 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1"} err="failed to get container status \"c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1\": rpc error: code = NotFound desc = could not find container \"c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1\": container with ID starting with c324e879cbdbf2a3fdde7122860e5f9f6d41b128cf1cb0de6fdc971343eff9d1 not found: ID does not exist" Oct 29 22:39:11.200934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.200862 5367 scope.go:122] "RemoveContainer" containerID="32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10" Oct 29 22:39:11.205906 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.204919 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-8cn6j"] Oct 29 22:39:11.208990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.207564 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.208990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.207640 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.208990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.207681 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.226372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.226347 5367 scope.go:122] "RemoveContainer" containerID="32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10" Oct 29 22:39:11.229552 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:11.228075 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10\": container with ID starting with 32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10 not found: ID does not exist" containerID="32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10" Oct 29 22:39:11.229552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.228111 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10"} err="failed to get container status \"32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10\": rpc error: code = NotFound desc = could not find container \"32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10\": container with ID starting with 32fc36751060020757018eefb5e6b50b31bb5437cee1a74caff676270e92bf10 not found: ID does not exist" Oct 29 22:39:11.237647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.237624 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103bf5a5-c7de-452c-9c7f-90f83e12f0bf" path="/var/lib/kubelet/pods/103bf5a5-c7de-452c-9c7f-90f83e12f0bf/volumes" Oct 29 22:39:11.238945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.238918 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff1f6bc-160e-4ab6-815d-c344cd149a65" path="/var/lib/kubelet/pods/aff1f6bc-160e-4ab6-815d-c344cd149a65/volumes" Oct 29 22:39:11.240064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.240047 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3345ff1-2c5e-47c5-856a-2caac6909907" path="/var/lib/kubelet/pods/d3345ff1-2c5e-47c5-856a-2caac6909907/volumes" Oct 29 22:39:11.245220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.245197 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt"] Oct 29 22:39:11.253649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.253628 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-6krwt"] Oct 29 22:39:11.285366 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:11.285333 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f9fd72_e4e7_425b_bd9d_2057366f47da.slice/crio-fdd96f65d3805dc406926586f89f33c337e553ce56e862ac27f032ac9a3d63bd WatchSource:0}: Error finding container fdd96f65d3805dc406926586f89f33c337e553ce56e862ac27f032ac9a3d63bd: Status 404 returned error can't find the container with id fdd96f65d3805dc406926586f89f33c337e553ce56e862ac27f032ac9a3d63bd Oct 29 22:39:11.286159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.286131 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6"] Oct 29 22:39:11.289720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.289653 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.289827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.289746 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.289827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.289793 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.317909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.315522 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:11.473405 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:11.473346 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd284d9a_851f_48c8_a365_b456bffc3149.slice/crio-ec49c17bc5ab83965268407a0f944c5867592efed7df76e5463ab34e7cb87c95 WatchSource:0}: Error finding container ec49c17bc5ab83965268407a0f944c5867592efed7df76e5463ab34e7cb87c95: Status 404 returned error can't find the container with id ec49c17bc5ab83965268407a0f944c5867592efed7df76e5463ab34e7cb87c95 Oct 29 22:39:11.478358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.478247 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.478726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.478532 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.478726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.478592 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.521998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.521970 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp"] Oct 29 22:39:11.708670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.708602 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br"] Oct 29 22:39:11.713652 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:11.713613 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e7493c_6aaa_4d36_b404_d806b0c7d1cf.slice/crio-1184dab3b7fcd6c1227c1be6149a1c7e3e803ecfbfffba192833ed8a8602b7d7 WatchSource:0}: Error finding container 1184dab3b7fcd6c1227c1be6149a1c7e3e803ecfbfffba192833ed8a8602b7d7: Status 404 returned error can't find the container with id 1184dab3b7fcd6c1227c1be6149a1c7e3e803ecfbfffba192833ed8a8602b7d7 Oct 29 22:39:11.718939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.718409 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.718939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.718500 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.718939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.718543 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:11.867726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.862092 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:11.867726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:11.862149 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:12.028540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.028434 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" event={"ID":"35f9fd72-e4e7-425b-bd9d-2057366f47da","Type":"ContainerStarted","Data":"6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c"} Oct 29 22:39:12.028540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.028529 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" event={"ID":"35f9fd72-e4e7-425b-bd9d-2057366f47da","Type":"ContainerStarted","Data":"fdd96f65d3805dc406926586f89f33c337e553ce56e862ac27f032ac9a3d63bd"} Oct 29 22:39:12.031847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.031814 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" event={"ID":"afb9022c-2c3f-43f4-a4e0-1ad32b282775","Type":"ContainerStarted","Data":"b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea"} Oct 29 22:39:12.032060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.032036 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" event={"ID":"afb9022c-2c3f-43f4-a4e0-1ad32b282775","Type":"ContainerStarted","Data":"211f03a6c96e319509e934cafe260e9b61cbce0b8b2160ae786c5d29af8b7d5f"} Oct 29 22:39:12.034172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.034150 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" event={"ID":"bd284d9a-851f-48c8-a365-b456bffc3149","Type":"ContainerStarted","Data":"8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764"} Oct 29 22:39:12.034326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.034313 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" event={"ID":"bd284d9a-851f-48c8-a365-b456bffc3149","Type":"ContainerStarted","Data":"ec49c17bc5ab83965268407a0f944c5867592efed7df76e5463ab34e7cb87c95"} Oct 29 22:39:12.036387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.036370 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:12.037986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.037965 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.038068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.038012 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:12.041315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.041293 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" event={"ID":"17e7493c-6aaa-4d36-b404-d806b0c7d1cf","Type":"ContainerStarted","Data":"c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452"} Oct 29 22:39:12.041379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.041339 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" event={"ID":"17e7493c-6aaa-4d36-b404-d806b0c7d1cf","Type":"ContainerStarted","Data":"1184dab3b7fcd6c1227c1be6149a1c7e3e803ecfbfffba192833ed8a8602b7d7"} Oct 29 22:39:12.090723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.083780 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podStartSLOduration=8.083758712 podStartE2EDuration="8.083758712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:12.072740044 +0000 UTC m=+1418.966814965" watchObservedRunningTime="2026-10-29 22:39:12.083758712 +0000 UTC m=+1418.977833634" Oct 29 22:39:12.121829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.121744 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podStartSLOduration=8.121724183 podStartE2EDuration="8.121724183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:12.119674743 +0000 UTC m=+1419.013749666" watchObservedRunningTime="2026-10-29 22:39:12.121724183 +0000 UTC m=+1419.015799105" Oct 29 22:39:12.172765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.172680 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podStartSLOduration=8.172658638 podStartE2EDuration="8.172658638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:12.166263292 +0000 UTC m=+1419.060338215" watchObservedRunningTime="2026-10-29 22:39:12.172658638 +0000 UTC m=+1419.066733560" Oct 29 22:39:12.504722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.466108 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:12.504722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.494873 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.504722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.494952 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:12.590064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.590005 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podStartSLOduration=9.589985346 podStartE2EDuration="9.589985346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:12.227139572 +0000 UTC m=+1419.121214518" watchObservedRunningTime="2026-10-29 22:39:12.589985346 +0000 UTC m=+1419.484060268" Oct 29 22:39:12.768120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.767528 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:12.768120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.767727 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.768120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.767773 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:12.814730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.814670 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:12.839902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.839502 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.839902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.839568 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:12.867497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.867340 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.867497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:12.867407 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:13.043786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.040187 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.043786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.040247 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:13.215942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.215869 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:13.236005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.235956 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:13.447723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.441797 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:13.447723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.441945 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 29 22:39:13.456144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.456096 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:13.456845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.456615 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:13.506249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.506200 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:13.506538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.506250 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:39:13.506694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.506292 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:39:13.787778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.731582 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8497a33f-5655-4a84-aa30-b98c024e9645" path="/var/lib/kubelet/pods/8497a33f-5655-4a84-aa30-b98c024e9645/volumes" Oct 29 22:39:13.932722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.819206 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.932722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.924708 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:13.932722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.863665 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.932722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.924829 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:13.932722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.865965 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.932722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.924932 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:13.973913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:13.973871 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.097916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.097877 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:14.098216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.096792 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.184140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.184101 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.184434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.184402 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:14.185883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.185838 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.210802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.206791 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.339023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.338987 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.339351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.339282 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:14.620550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.620508 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.635412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.630487 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.635768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.635745 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:39:14.708754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.704789 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.708754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.705154 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:39:14.708754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.705448 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:39:14.708754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.705797 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:39:14.708754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.705857 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:39:14.708754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.705893 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:39:14.745784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.745246 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.747079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.747044 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:14.760962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.760938 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:39:14.763556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.761175 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:39:14.772031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.761206 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:39:14.772880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.772862 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:39:14.773055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.773042 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:39:14.786495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.774927 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.786495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.774970 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:14.789883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.789863 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:39:14.810649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.810617 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.810926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.810898 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:14.865121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.865087 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.865372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:14.865345 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:15.117237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.101802 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:15.117237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.116838 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.248356 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.248414 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.272421 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.324834 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.350594 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": context deadline exceeded" Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.350922 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.367400 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.387237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.367453 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:15.637332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.637239 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:39:15.665853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.662775 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:39:15.666153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.666134 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:39:15.720728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.716513 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:15.720728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.716560 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:15.727371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.726794 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:39:15.746919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.746875 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:15.759071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.759040 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:39:15.783103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.772813 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:15.797278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.788938 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:15.849738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.800815 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:39:15.849738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.831256 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.849738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.831310 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:15.849738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.839844 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.849738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.839893 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:15.877221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.877186 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.879236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:15.879196 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:16.174722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.104854 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:16.174722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.104924 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:16.174722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.132573 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.295725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.279768 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.295725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.290838 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.318721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.316929 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.356920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.356877 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.542772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.515878 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:16.621722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.621225 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:16.687866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.687821 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.735723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.718070 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:39:16.735723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.733276 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.735723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.733534 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.762813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.762768 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:39:16.864007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.796291 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:16.872017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.871986 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:16.872276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.872251 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:16.960681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.960651 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 29 22:39:16.960995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.960950 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 29 22:39:16.983722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.961220 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:16.984203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.984161 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:16.984400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:16.960543 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:17.068725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.044731 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:17.072337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.070861 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:17.072337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.070920 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:17.162308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.160946 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:39:17.316737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.307909 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:17.317351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.316996 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:17.317351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.317046 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:17.334315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.334278 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:39:17.367308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.367268 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:39:17.749171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.749127 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:17.749627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.749596 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:17.749751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.749654 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:17.749830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.749789 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:39:17.758051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.758015 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:39:17.763960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.763927 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:39:17.827761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.809169 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:39:17.844830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.809842 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:17.845136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.845106 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:17.867978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:17.846723 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:18.106793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.106757 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:18.107095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.107054 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:18.137979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.137934 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:39:18.210900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.210857 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:39:18.343835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.343778 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:39:18.344992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.344958 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:39:18.406791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.366393 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" podUID="fd4be442-0fd5-483a-a2fb-45561dfdaafc" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:18.406791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.389908 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 29 22:39:18.406791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.405798 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:39:18.483823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.483771 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:39:18.514886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.514846 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:18.515167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.515140 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:18.592333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.592298 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:39:18.592599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.592582 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl"] Oct 29 22:39:18.592812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.592798 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:39:18.619721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.599818 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:39:18.645743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.645714 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:39:18.646082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.646064 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:39:18.747128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.737996 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" podUID="fd4be442-0fd5-483a-a2fb-45561dfdaafc" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:18.807146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.757287 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:18.807146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.759406 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:39:18.807146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.760229 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:18.807146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.760491 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": context deadline exceeded" Oct 29 22:39:18.849759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.844929 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:39:18.851853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.851816 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:18.860720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.852040 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:18.898361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.870676 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:18.898361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:18.870744 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:19.033781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.017208 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.106728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.094636 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:19.167151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.167114 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a17f80-2zhzl"] Oct 29 22:39:19.223726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.221032 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.241718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.240812 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.330720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.324002 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:19.331050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.331016 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:19.494580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.359464 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:19.567926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.360185 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.581632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.360209 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:39:19.600722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.583768 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.600722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.491520 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:39:19.600722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.407785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.602598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.601464 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.681655 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.801987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.795324 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:39:19.801987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.795398 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 22:39:19.801987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.795931 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.801987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.798169 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:39:19.815201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.802519 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:19.871780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.848363 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:39:19.873842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.873812 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:39:19.894921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.894823 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 22:39:19.895662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:19.880509 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4" path="/var/lib/kubelet/pods/b4d3de3a-89ef-4e75-8cf6-5ffffd1115a4/volumes" Oct 29 22:39:20.042851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.041461 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:39:20.118722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.095828 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:20.118722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.095890 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:20.344722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.336162 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.344722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.337724 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:20.344722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.337764 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:20.359816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.359774 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.546803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.546763 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:39:20.595789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.595667 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.596461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.596326 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.597007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.596984 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:39:20.623725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.614175 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.637065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.636022 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:39:20.693045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.693015 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:39:20.838711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.836131 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:39:20.838711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.837497 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.838974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.838956 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:39:20.840682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.840168 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.871349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.871215 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:20.891729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.891446 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:39:20.923667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.923625 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:20.925288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.925266 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:20.940727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:20.939884 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:21.002872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.001795 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:21.019894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.006446 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:39:21.022416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.020466 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:21.027941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.027915 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:39:21.050994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.050963 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:39:21.051212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.051185 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:39:21.114941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.114907 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:39:21.211317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.211235 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:39:21.321488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.321454 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:39:21.321675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.321539 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:39:21.321675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.321590 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:39:21.321675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.321639 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:39:21.339362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.339330 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 29 22:39:21.340922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.340890 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 29 22:39:21.341414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:21.339446 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:39:22.043256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:22.043227 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:22.043807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:22.043783 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:22.044497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:22.044475 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 22:39:22.323164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:22.323137 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:22.323795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:22.323765 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:22.324530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:22.324503 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:39:22.339232 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:22.339194 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:39:25.030442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:25.030410 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5"] Oct 29 22:39:25.042937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:25.042912 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2c9ba9-7rvm5"] Oct 29 22:39:25.224918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:25.224887 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b274dc39-92a7-4a00-894f-57b6314671c4" path="/var/lib/kubelet/pods/b274dc39-92a7-4a00-894f-57b6314671c4/volumes" Oct 29 22:39:35.144038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.144002 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9"] Oct 29 22:39:35.144493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.144235 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" containerID="cri-o://ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84" gracePeriod=30 Oct 29 22:39:35.169750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.168298 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx"] Oct 29 22:39:35.169750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.168540 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" containerID="cri-o://7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370" gracePeriod=30 Oct 29 22:39:35.208402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.208372 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb"] Oct 29 22:39:35.209088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.209052 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" containerID="cri-o://4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35" gracePeriod=30 Oct 29 22:39:35.236639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.236608 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm"] Oct 29 22:39:35.237149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.237100 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" containerID="cri-o://0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad" gracePeriod=30 Oct 29 22:39:35.644434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.644403 5367 scope.go:122] "RemoveContainer" containerID="b1496748da4296847a5f9254a8bd6f5fa0550d0ffdd45059dcd93c7d31782b13" Oct 29 22:39:35.680749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.680720 5367 scope.go:122] "RemoveContainer" containerID="ae1aa5b6773aa3a07d0f4f1da4e375657a8df3c5d5ae10c7e8bc9fa723808a51" Oct 29 22:39:35.695826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.695728 5367 scope.go:122] "RemoveContainer" containerID="8457c2b33e791b9e62eb55ff189e26dafb870966e853b284c2ca52498d5e4e23" Oct 29 22:39:35.706063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.706042 5367 scope.go:122] "RemoveContainer" containerID="70aba5a8871acc05ab23d52d1dbaff9f8562f3ea3c2f6dec57aa70fd6cfb6503" Oct 29 22:39:35.716886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.716868 5367 scope.go:122] "RemoveContainer" containerID="df859a437eceb0aee86757902a848437d0a39ee3c3580a4f8617131e0085fae5" Oct 29 22:39:35.728533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.728512 5367 scope.go:122] "RemoveContainer" containerID="8d75e2f79be790d656f9797394ae55347db190f1749ca96cd2dac66e3f7ff8ea" Oct 29 22:39:35.792620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.792589 5367 scope.go:122] "RemoveContainer" containerID="e3e97d9fefd88c5f9661ea1bbdfd3c96fb32a3bbc5373d26089a5323917a29fa" Oct 29 22:39:35.806607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.806582 5367 scope.go:122] "RemoveContainer" containerID="afa4ef1d9dcdca984c1f5729f70f4ef9a12761871bd99c67d694d4cb5502d2c3" Oct 29 22:39:35.818477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:35.818456 5367 scope.go:122] "RemoveContainer" containerID="dd60f0f656fb17a2198696e8ea7a244e5a4586b404ce15292bf268f845add0a0" Oct 29 22:39:40.560116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.560083 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644040 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644115 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644185 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644234 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644272 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644375 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644516 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644571 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.644661 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket\") pod \"357cc0bd-e32e-48f4-9caf-f901888c530f\" (UID: \"357cc0bd-e32e-48f4-9caf-f901888c530f\") " Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.645906 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.646230 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:40.647804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.646782 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.652814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.650911 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.652814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.651104 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.652814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.651269 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.653565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.653457 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.653565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.653523 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:40.654408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.654366 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6" pod "357cc0bd-e32e-48f4-9caf-f901888c530f" (UID: "357cc0bd-e32e-48f4-9caf-f901888c530f"). InnerVolumeSpecName "kube-api-access-6b2r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.662281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.662250 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4"] Oct 29 22:39:40.663547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.663523 5367 cpu_manager.go:414] "containerMap: removing container" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" Oct 29 22:39:40.663547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.663549 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" Oct 29 22:39:40.663829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.663813 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerName="istio-proxy" Oct 29 22:39:40.668716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.668673 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.685537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.685369 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4"] Oct 29 22:39:40.690752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.690350 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-433712-openshift-default-dockercfg-lnlp8\"" Oct 29 22:39:40.702481 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.702459 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:39:40.708486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.708464 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:39:40.713071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.713050 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:39:40.751807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.751774 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.752108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.752084 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.752307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.752289 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.752467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.752452 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.762124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762044 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.762249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762151 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.762249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762181 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.762368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762303 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.762492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762474 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktqg\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.762669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762656 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762675 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357cc0bd-e32e-48f4-9caf-f901888c530f-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762688 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762716 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762729 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762740 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762754 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.762771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762766 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357cc0bd-e32e-48f4-9caf-f901888c530f-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.763119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.762779 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b2r6\" (UniqueName: \"kubernetes.io/projected/357cc0bd-e32e-48f4-9caf-f901888c530f-kube-api-access-6b2r6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.790061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.790029 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp"] Oct 29 22:39:40.791398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.791372 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" Oct 29 22:39:40.791563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.791547 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" Oct 29 22:39:40.791720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.791689 5367 cpu_manager.go:414] "containerMap: removing container" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" Oct 29 22:39:40.791843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.791829 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" Oct 29 22:39:40.792020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.792008 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" Oct 29 22:39:40.792130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.792110 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" Oct 29 22:39:40.792356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.792339 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" Oct 29 22:39:40.792446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.792364 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerName="istio-proxy" Oct 29 22:39:40.792446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.792380 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="168026fd-c56f-432c-ab83-85323cdda342" containerName="istio-proxy" Oct 29 22:39:40.797081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.797061 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.813379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.813342 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c06e05-openshift-default-dockercfg-89tnf\"" Oct 29 22:39:40.818693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.818657 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp"] Oct 29 22:39:40.837758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.836520 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb"] Oct 29 22:39:40.841949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.841929 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:40.853447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.853048 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb"] Oct 29 22:39:40.863776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.860924 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:39:40.863776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.861869 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:39:40.864036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864010 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864097 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864144 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864204 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864233 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864279 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.864329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864318 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.864496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864364 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.864496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864389 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.864496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864421 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.864496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864448 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864512 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.864725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864574 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864629 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.864725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864657 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.864940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864807 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.864940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864869 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.865045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864951 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.865045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.864998 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket\") pod \"168026fd-c56f-432c-ab83-85323cdda342\" (UID: \"168026fd-c56f-432c-ab83-85323cdda342\") " Oct 29 22:39:40.865145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865067 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.865145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865114 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.865257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865182 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.865257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865217 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.865352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865264 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.865352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865317 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.865462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865366 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo\") pod \"1f3a0dc5-300a-47f9-925a-939895679004\" (UID: \"1f3a0dc5-300a-47f9-925a-939895679004\") " Oct 29 22:39:40.865462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865449 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy\") pod \"f31010b6-2721-4990-a044-68d8c9f0ccaf\" (UID: \"f31010b6-2721-4990-a044-68d8c9f0ccaf\") " Oct 29 22:39:40.865985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.865954 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktqg\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866142 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866231 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866319 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866411 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866451 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866527 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866545 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.866660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.866653 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.868135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.868106 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:40.868470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.868447 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.869736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.869666 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.871471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.871109 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.871471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.871256 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.872728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.872584 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:40.872832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.872759 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.873068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.873039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.873211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.873160 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.873787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.873284 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.873787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.873561 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.873941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.873853 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.874668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.874062 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.874668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.873996 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:40.874668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.874129 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.874668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.874328 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.874668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.874403 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:40.875001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.874907 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.875218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.875103 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.875357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.875334 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.875505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.875476 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.875582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.875533 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.876323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.876014 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp" pod "168026fd-c56f-432c-ab83-85323cdda342" (UID: "168026fd-c56f-432c-ab83-85323cdda342"). InnerVolumeSpecName "kube-api-access-bqnmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.876323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.876256 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.876502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.876416 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.877211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.877140 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:40.877362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.877343 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.878035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.878009 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:40.880902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.880879 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.881167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.881131 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "kube-api-access-pfdp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.881554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.881355 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.881554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.881523 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.881938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.881808 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86"] Oct 29 22:39:40.882778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.882749 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy" pod "1f3a0dc5-300a-47f9-925a-939895679004" (UID: "1f3a0dc5-300a-47f9-925a-939895679004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:40.883924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.883861 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2" pod "f31010b6-2721-4990-a044-68d8c9f0ccaf" (UID: "f31010b6-2721-4990-a044-68d8c9f0ccaf"). InnerVolumeSpecName "kube-api-access-pnxj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:40.890564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.890514 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktqg\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.890921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.890904 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.902977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.902945 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-xvvt4\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:40.907907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.907838 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86"] Oct 29 22:39:40.913862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.913480 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 22:39:40.913862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.913565 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-e526ec-serving-cert\"" Oct 29 22:39:40.913862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.913656 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:39:40.977943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.977898 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.978278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.978258 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.978488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.978470 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpggp\" (UniqueName: \"kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:40.979003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.978977 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.979245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.979192 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.979245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.979237 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjzq\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.979503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.979264 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.979503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.979404 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.979503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.979497 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.980751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.980673 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:40.980886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.980762 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.981033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.980958 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.981287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981266 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.981459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981441 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.981587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981572 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.981814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981795 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:40.982086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981864 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.982086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981894 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhbf\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.982086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981936 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:40.982086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.981985 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.982086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982120 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982257 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168026fd-c56f-432c-ab83-85323cdda342-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982274 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982287 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdp2\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-kube-api-access-pfdp2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982303 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982317 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982330 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982343 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982361 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982375 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982469 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168026fd-c56f-432c-ab83-85323cdda342-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982487 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982500 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxj2\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-kube-api-access-pnxj2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982514 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982531 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.982561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982544 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3a0dc5-300a-47f9-925a-939895679004-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982596 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982625 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982639 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3a0dc5-300a-47f9-925a-939895679004-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982652 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982664 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31010b6-2721-4990-a044-68d8c9f0ccaf-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982676 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982689 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3a0dc5-300a-47f9-925a-939895679004-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982767 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3a0dc5-300a-47f9-925a-939895679004-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.982786 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31010b6-2721-4990-a044-68d8c9f0ccaf-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.983653 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.983671 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqnmp\" (UniqueName: \"kubernetes.io/projected/168026fd-c56f-432c-ab83-85323cdda342-kube-api-access-bqnmp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.984664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.983690 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168026fd-c56f-432c-ab83-85323cdda342-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:39:40.992472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:40.992452 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:41.010123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.010089 5367 generic.go:356] "Generic (PLEG): container finished" podID="1f3a0dc5-300a-47f9-925a-939895679004" containerID="7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370" exitCode=0 Oct 29 22:39:41.010246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.010136 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" Oct 29 22:39:41.010246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.010167 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" event={"ID":"1f3a0dc5-300a-47f9-925a-939895679004","Type":"ContainerDied","Data":"7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370"} Oct 29 22:39:41.010246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.010235 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" event={"ID":"1f3a0dc5-300a-47f9-925a-939895679004","Type":"ContainerDied","Data":"250004dd30544ce608dbb735736ddcfecd2c6b6fa95c50256b5d15bbf6089c46"} Oct 29 22:39:41.010407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.010251 5367 scope.go:122] "RemoveContainer" containerID="7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370" Oct 29 22:39:41.012539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.012519 5367 generic.go:356] "Generic (PLEG): container finished" podID="357cc0bd-e32e-48f4-9caf-f901888c530f" containerID="ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84" exitCode=0 Oct 29 22:39:41.012671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.012601 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" event={"ID":"357cc0bd-e32e-48f4-9caf-f901888c530f","Type":"ContainerDied","Data":"ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84"} Oct 29 22:39:41.012671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.012638 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" event={"ID":"357cc0bd-e32e-48f4-9caf-f901888c530f","Type":"ContainerDied","Data":"751a0c0192e1a0220fc9807bb733687e7efb0eb78320b63333c2478a2bd5eb06"} Oct 29 22:39:41.012862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.012691 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9" Oct 29 22:39:41.027929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.027907 5367 scope.go:122] "RemoveContainer" containerID="7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370" Oct 29 22:39:41.028390 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:41.028369 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370\": container with ID starting with 7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370 not found: ID does not exist" containerID="7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370" Oct 29 22:39:41.028496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.028396 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370"} err="failed to get container status \"7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370\": rpc error: code = NotFound desc = could not find container \"7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370\": container with ID starting with 7f3a201ff20c8666a3bf2c5d9d90a994abff302b6a3bfba8b7e68c97d1777370 not found: ID does not exist" Oct 29 22:39:41.028496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.028414 5367 scope.go:122] "RemoveContainer" containerID="ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84" Oct 29 22:39:41.029333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.029309 5367 generic.go:356] "Generic (PLEG): container finished" podID="f31010b6-2721-4990-a044-68d8c9f0ccaf" containerID="4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35" exitCode=0 Oct 29 22:39:41.029426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.029375 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" event={"ID":"f31010b6-2721-4990-a044-68d8c9f0ccaf","Type":"ContainerDied","Data":"4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35"} Oct 29 22:39:41.029498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.029438 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" event={"ID":"f31010b6-2721-4990-a044-68d8c9f0ccaf","Type":"ContainerDied","Data":"d1c77592d6b707532ece3dd257852dd778d23654318e0d9cc6bd4dd1447e893f"} Oct 29 22:39:41.029826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.029811 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb" Oct 29 22:39:41.034721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.032951 5367 generic.go:356] "Generic (PLEG): container finished" podID="168026fd-c56f-432c-ab83-85323cdda342" containerID="0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad" exitCode=0 Oct 29 22:39:41.034721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.033008 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" event={"ID":"168026fd-c56f-432c-ab83-85323cdda342","Type":"ContainerDied","Data":"0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad"} Oct 29 22:39:41.034721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.033044 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" event={"ID":"168026fd-c56f-432c-ab83-85323cdda342","Type":"ContainerDied","Data":"7f6b5fbe2fdddfa418b80cfcc9c6ade7b2453609cc142d67695434b99cb219de"} Oct 29 22:39:41.034721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.033103 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm" Oct 29 22:39:41.044130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.044066 5367 scope.go:122] "RemoveContainer" containerID="ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84" Oct 29 22:39:41.044429 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:41.044409 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84\": container with ID starting with ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84 not found: ID does not exist" containerID="ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84" Oct 29 22:39:41.044519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.044440 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84"} err="failed to get container status \"ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84\": rpc error: code = NotFound desc = could not find container \"ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84\": container with ID starting with ae02b48bf1e1fd6f8d00d24f193ab6cdc53d091ece2b3e7416c0d9ec39ee2e84 not found: ID does not exist" Oct 29 22:39:41.044519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.044463 5367 scope.go:122] "RemoveContainer" containerID="4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35" Oct 29 22:39:41.070448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.070422 5367 scope.go:122] "RemoveContainer" containerID="4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35" Oct 29 22:39:41.070912 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:41.070875 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35\": container with ID starting with 4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35 not found: ID does not exist" containerID="4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35" Oct 29 22:39:41.071027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.070923 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35"} err="failed to get container status \"4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35\": rpc error: code = NotFound desc = could not find container \"4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35\": container with ID starting with 4f01163850938ac3d2ff276d5514d2b21ae8d8e5fa9ff16a7a999d6e5985df35 not found: ID does not exist" Oct 29 22:39:41.071027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.070948 5367 scope.go:122] "RemoveContainer" containerID="0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad" Oct 29 22:39:41.089054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.088070 5367 scope.go:122] "RemoveContainer" containerID="0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad" Oct 29 22:39:41.089054 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:41.088393 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad\": container with ID starting with 0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad not found: ID does not exist" containerID="0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad" Oct 29 22:39:41.089054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.088424 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad"} err="failed to get container status \"0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad\": rpc error: code = NotFound desc = could not find container \"0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad\": container with ID starting with 0a38b74a5c3f409a180e1ccb535e7b53dc58396b4573724edf4e8b63851f52ad not found: ID does not exist" Oct 29 22:39:41.089054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089045 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.089301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089075 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.089301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089111 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.089301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089148 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpggp\" (UniqueName: \"kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.089301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089215 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.089301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089258 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.089301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089294 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjzq\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.089613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089324 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089736 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089824 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089895 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089938 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.089988 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090059 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090075 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090106 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090133 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090210 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090276 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090287 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090318 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhbf\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090364 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090427 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.090468 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.091229 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.091597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.091434 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.092624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.092056 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.094520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.092691 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.094520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.092955 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.094520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.093193 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.094520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.094282 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.094520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.094362 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.095238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.095199 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.096318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.096292 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.096720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.096647 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.097564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.097529 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.099726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.098826 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.100176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.100139 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.101332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.101301 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.113217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.113191 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjzq\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.113335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.113310 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpggp\" (UniqueName: \"kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp\") pod \"maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.116829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.116727 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb"] Oct 29 22:39:41.118891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.118835 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhbf\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.119463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.119410 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.119761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.119680 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token\") pod \"e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.122212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.122190 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-28gxb"] Oct 29 22:39:41.160716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.160665 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm"] Oct 29 22:39:41.167497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.167468 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-rwpgm"] Oct 29 22:39:41.171688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.171662 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:41.222928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.222892 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:41.246893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.246856 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168026fd-c56f-432c-ab83-85323cdda342" path="/var/lib/kubelet/pods/168026fd-c56f-432c-ab83-85323cdda342/volumes" Oct 29 22:39:41.247839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.247803 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31010b6-2721-4990-a044-68d8c9f0ccaf" path="/var/lib/kubelet/pods/f31010b6-2721-4990-a044-68d8c9f0ccaf/volumes" Oct 29 22:39:41.248607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.248308 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9"] Oct 29 22:39:41.248607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.248344 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-c28g9"] Oct 29 22:39:41.256152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.256129 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx"] Oct 29 22:39:41.265854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.265623 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx"] Oct 29 22:39:41.448418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.448386 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4"] Oct 29 22:39:41.450587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.448902 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:41.564976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.564877 5367 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:39:41.568405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.568354 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-99whx" podUID="1f3a0dc5-300a-47f9-925a-939895679004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:41.629434 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:41.629375 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb7a0b4_b57d_4d5d_b7c7_ff89ccc5f3a3.slice/crio-bba25861367dd1bf748a83d7a6754ce7b638e687b721ed1e3a1f45adab70d866 WatchSource:0}: Error finding container bba25861367dd1bf748a83d7a6754ce7b638e687b721ed1e3a1f45adab70d866: Status 404 returned error can't find the container with id bba25861367dd1bf748a83d7a6754ce7b638e687b721ed1e3a1f45adab70d866 Oct 29 22:39:41.648291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.640451 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:41.648291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.640616 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:41.648291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.640687 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:41.865667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.862511 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:39:41.873229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.873198 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb"] Oct 29 22:39:41.968488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.968453 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86"] Oct 29 22:39:41.968847 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:41.968822 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583be168_6bc5_4983_a98c_9bca968b4b97.slice/crio-bbce4aaf4f807da225a327ede2677035cedf2667ed0b172feb36e4594c5ef28e WatchSource:0}: Error finding container bbce4aaf4f807da225a327ede2677035cedf2667ed0b172feb36e4594c5ef28e: Status 404 returned error can't find the container with id bbce4aaf4f807da225a327ede2677035cedf2667ed0b172feb36e4594c5ef28e Oct 29 22:39:41.972634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.972597 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:41.972786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.972678 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:41.972786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:41.972734 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:42.052251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.052211 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" event={"ID":"da53d7be-436c-4373-9572-4fa61531175d","Type":"ContainerStarted","Data":"9ae7b1444d430ba9efc27c6a7992d92802e0e7165e9e0d429fd468a460a9fefc"} Oct 29 22:39:42.056161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.056124 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" event={"ID":"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3","Type":"ContainerStarted","Data":"793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a"} Oct 29 22:39:42.056306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.056182 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" event={"ID":"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3","Type":"ContainerStarted","Data":"bba25861367dd1bf748a83d7a6754ce7b638e687b721ed1e3a1f45adab70d866"} Oct 29 22:39:42.062005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.061974 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" event={"ID":"583be168-6bc5-4983-a98c-9bca968b4b97","Type":"ContainerStarted","Data":"bbce4aaf4f807da225a327ede2677035cedf2667ed0b172feb36e4594c5ef28e"} Oct 29 22:39:42.114644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.114613 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp"] Oct 29 22:39:42.117161 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:42.117129 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be234cd_119e_42c7_918d_03b3259114b4.slice/crio-f4a76d8d9d0d7c8aff2e4be524544cb188eb7de32958c9a0e283c8cdd49f3b90 WatchSource:0}: Error finding container f4a76d8d9d0d7c8aff2e4be524544cb188eb7de32958c9a0e283c8cdd49f3b90: Status 404 returned error can't find the container with id f4a76d8d9d0d7c8aff2e4be524544cb188eb7de32958c9a0e283c8cdd49f3b90 Oct 29 22:39:42.117358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.117305 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podStartSLOduration=37.117287866 podStartE2EDuration="37.117287866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:42.10718024 +0000 UTC m=+1449.001255162" watchObservedRunningTime="2026-10-29 22:39:42.117287866 +0000 UTC m=+1449.011362789" Oct 29 22:39:42.128013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.124967 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:42.128013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.125058 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:42.128013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.125103 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:42.993119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:42.993045 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:43.000540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.000184 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:43.101659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.101623 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" event={"ID":"6be234cd-119e-42c7-918d-03b3259114b4","Type":"ContainerStarted","Data":"7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c"} Oct 29 22:39:43.101872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.101679 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" event={"ID":"6be234cd-119e-42c7-918d-03b3259114b4","Type":"ContainerStarted","Data":"f4a76d8d9d0d7c8aff2e4be524544cb188eb7de32958c9a0e283c8cdd49f3b90"} Oct 29 22:39:43.108319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.108223 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" event={"ID":"583be168-6bc5-4983-a98c-9bca968b4b97","Type":"ContainerStarted","Data":"dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b"} Oct 29 22:39:43.108319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.108284 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:43.110030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.110005 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:39:43.150453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.150376 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podStartSLOduration=30.150361138 podStartE2EDuration="30.150361138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:43.149044967 +0000 UTC m=+1450.043119892" watchObservedRunningTime="2026-10-29 22:39:43.150361138 +0000 UTC m=+1450.044436060" Oct 29 22:39:43.207164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.207106 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podStartSLOduration=19.207090983 podStartE2EDuration="19.207090983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:43.200653913 +0000 UTC m=+1450.094728834" watchObservedRunningTime="2026-10-29 22:39:43.207090983 +0000 UTC m=+1450.101165904" Oct 29 22:39:43.224862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.224829 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 29 22:39:43.225020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.224900 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 29 22:39:43.235280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.233846 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3a0dc5-300a-47f9-925a-939895679004" path="/var/lib/kubelet/pods/1f3a0dc5-300a-47f9-925a-939895679004/volumes" Oct 29 22:39:43.235280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.234626 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357cc0bd-e32e-48f4-9caf-f901888c530f" path="/var/lib/kubelet/pods/357cc0bd-e32e-48f4-9caf-f901888c530f/volumes" Oct 29 22:39:43.235280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.235226 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:43.490572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.490541 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:43.497635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:43.497598 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:44.117182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:44.117072 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:44.118154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:44.118128 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:39:44.229073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:44.224374 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 29 22:39:44.229073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:44.224434 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 29 22:39:44.336818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:44.336788 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:39:44.336972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:44.336833 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:45.224913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:45.224877 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 29 22:39:45.225343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:45.224937 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 29 22:39:46.224163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:46.224131 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 29 22:39:46.224388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:46.224178 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 29 22:39:47.247057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.246517 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 29 22:39:47.247057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.246629 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 29 22:39:47.326598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.326565 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56"] Oct 29 22:39:47.331918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.331895 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56"] Oct 29 22:39:47.332042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.332008 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.370892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.361924 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-433712-dockercfg-pg6rt\"" Oct 29 22:39:47.370892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.361991 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-433712\"" Oct 29 22:39:47.505596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.505554 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl5z\" (UniqueName: \"kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z\") pod \"payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.505831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.505813 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.607517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.607478 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl5z\" (UniqueName: \"kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z\") pod \"payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.607717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.607637 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.608512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.608475 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.627525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.627503 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl5z\" (UniqueName: \"kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z\") pod \"payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:47.740946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:47.740851 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:48.087260 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:48.087226 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64910860_5fd8_4c44_8c90_15c9a17b4bd4.slice/crio-b650f30da31c65df1a465bb8787f5048fdd871a8028ada1b5b919afa555015b4 WatchSource:0}: Error finding container b650f30da31c65df1a465bb8787f5048fdd871a8028ada1b5b919afa555015b4: Status 404 returned error can't find the container with id b650f30da31c65df1a465bb8787f5048fdd871a8028ada1b5b919afa555015b4 Oct 29 22:39:48.087660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.087633 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56"] Oct 29 22:39:48.220930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.220899 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" event={"ID":"64910860-5fd8-4c44-8c90-15c9a17b4bd4","Type":"ContainerStarted","Data":"6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca"} Oct 29 22:39:48.221058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.220958 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:48.221058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.220981 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" event={"ID":"64910860-5fd8-4c44-8c90-15c9a17b4bd4","Type":"ContainerStarted","Data":"b650f30da31c65df1a465bb8787f5048fdd871a8028ada1b5b919afa555015b4"} Oct 29 22:39:48.222854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.222824 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" event={"ID":"da53d7be-436c-4373-9572-4fa61531175d","Type":"ContainerStarted","Data":"41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190"} Oct 29 22:39:48.222980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.222919 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:48.223675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.223649 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 29 22:39:48.223796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.223730 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 29 22:39:48.224820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.224799 5367 generic.go:356] "Generic (PLEG): container finished" podID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerID="4548dd87f251184c4db6b87237a44e023d49ea243c794942a2109c91512bb3c5" exitCode=0 Oct 29 22:39:48.224916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.224836 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" event={"ID":"722caee6-f913-4d87-a3b3-9342ccc42ebe","Type":"ContainerDied","Data":"4548dd87f251184c4db6b87237a44e023d49ea243c794942a2109c91512bb3c5"} Oct 29 22:39:48.225240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.225224 5367 scope.go:122] "RemoveContainer" containerID="4548dd87f251184c4db6b87237a44e023d49ea243c794942a2109c91512bb3c5" Oct 29 22:39:48.226956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.226929 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:48.227036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.227001 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:48.227077 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.227035 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:39:48.292018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.291962 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" podStartSLOduration=2.2919422799999998 podStartE2EDuration="2.29194228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:48.283874548 +0000 UTC m=+1455.177949471" watchObservedRunningTime="2026-10-29 22:39:48.29194228 +0000 UTC m=+1455.186017204" Oct 29 22:39:48.533391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.533332 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" podStartSLOduration=22.194144125 podStartE2EDuration="27.533313256s" totalImagesPullingTime="5.339169131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:48.391561942 +0000 UTC m=+1455.285636863" watchObservedRunningTime="2026-10-29 22:39:48.533313256 +0000 UTC m=+1455.427388178" Oct 29 22:39:48.534855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.534818 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj"] Oct 29 22:39:48.539020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.539001 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:48.555225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.555192 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c06e05\"" Oct 29 22:39:48.555595 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.555557 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c06e05-dockercfg-thz8p\"" Oct 29 22:39:48.566189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.566162 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj"] Oct 29 22:39:48.822521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.822475 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6nb\" (UniqueName: \"kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb\") pod \"payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:48.822958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.822917 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:48.926191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.926153 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6nb\" (UniqueName: \"kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb\") pod \"payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:48.926919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.926896 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:48.927663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.927639 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:48.946833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:48.946802 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6nb\" (UniqueName: \"kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb\") pod \"payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:49.042800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.042764 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6"] Oct 29 22:39:49.064984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.063035 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece9a38-cnjz6"] Oct 29 22:39:49.156629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.156534 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:49.267100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.267060 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b258e4-447d-4e0c-a956-756391bb785d" path="/var/lib/kubelet/pods/b0b258e4-447d-4e0c-a956-756391bb785d/volumes" Oct 29 22:39:49.268154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.268121 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" event={"ID":"722caee6-f913-4d87-a3b3-9342ccc42ebe","Type":"ContainerStarted","Data":"f234bf16f00f179fbc78421641f70fd7827ebf982082fe00073f24071f1d1e79"} Oct 29 22:39:49.321998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.321743 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:39:49.323579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.323549 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:39:49.323853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.323613 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:39:49.490274 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:39:49.490239 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b1a47b_add4_46a5_ae68_3c1d2f211ab8.slice/crio-f45a69656de4c65e8f55e22f04b93fafe6f4c05ee5f574a5698e3b67688f3a43 WatchSource:0}: Error finding container f45a69656de4c65e8f55e22f04b93fafe6f4c05ee5f574a5698e3b67688f3a43: Status 404 returned error can't find the container with id f45a69656de4c65e8f55e22f04b93fafe6f4c05ee5f574a5698e3b67688f3a43 Oct 29 22:39:49.491825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.491799 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj"] Oct 29 22:39:49.523888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.523858 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:49.524362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.524262 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:49.525349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:49.525330 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:39:50.288832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:50.288790 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" event={"ID":"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8","Type":"ContainerStarted","Data":"2819bccd276f214497de9e8feedff3ec92f14de9a64acf09e07eba1d311237c3"} Oct 29 22:39:50.289034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:50.288860 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:39:50.289034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:50.288883 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" event={"ID":"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8","Type":"ContainerStarted","Data":"f45a69656de4c65e8f55e22f04b93fafe6f4c05ee5f574a5698e3b67688f3a43"} Oct 29 22:39:50.321782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:50.321750 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:39:50.321947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:50.321810 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:39:51.321429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:51.321393 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:39:51.321871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:51.321444 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:39:52.321859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:52.321825 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:39:52.322412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:52.321879 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:39:52.333523 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:39:52.333489 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:39:53.322315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:53.322279 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:39:53.322863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:53.322350 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:39:54.333724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:54.331186 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:39:54.333724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:54.331259 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 29 22:39:54.349743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:54.345757 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:39:54.405952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:54.405896 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" podStartSLOduration=6.405882117 podStartE2EDuration="6.405882117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:50.331871425 +0000 UTC m=+1457.225946346" watchObservedRunningTime="2026-10-29 22:39:54.405882117 +0000 UTC m=+1461.299957037" Oct 29 22:39:55.327339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:55.327309 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:39:55.327822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:55.327805 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:39:55.328496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:55.328478 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:39:59.271691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:59.271661 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:39:59.864809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:39:59.864774 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg"] Oct 29 22:40:01.311583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:01.311547 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:40:04.865813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:04.865746 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerName="payload-pre-processing" containerID="cri-o://232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4" gracePeriod=25 Oct 29 22:40:12.132854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:12.132810 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:22.133938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:22.133898 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:22.335320 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:40:22.335283 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:40:25.456077 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.456042 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:40:25.594986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.594957 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume\") pod \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " Oct 29 22:40:25.595184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.595084 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt\" (UniqueName: \"kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt\") pod \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\" (UID: \"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0\") " Oct 29 22:40:25.595323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.595301 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume" pod "e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" (UID: "e1b8d2dd-4f61-48ee-9f67-482aea98f1d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:25.595741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.595725 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:25.597013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.596991 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt" pod "e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" (UID: "e1b8d2dd-4f61-48ee-9f67-482aea98f1d0"). InnerVolumeSpecName "kube-api-access-jkjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:25.697414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.697382 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjmt\" (UniqueName: \"kubernetes.io/projected/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0-kube-api-access-jkjmt\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:25.841850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.841816 5367 generic.go:356] "Generic (PLEG): container finished" podID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerID="232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4" exitCode=0 Oct 29 22:40:25.842042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.841861 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" Oct 29 22:40:25.842042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.841898 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" event={"ID":"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0","Type":"ContainerDied","Data":"232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4"} Oct 29 22:40:25.842042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.841952 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg" event={"ID":"e1b8d2dd-4f61-48ee-9f67-482aea98f1d0","Type":"ContainerDied","Data":"6711b7b35041a639ff729e3ea0f9c47b120827771ca6bbda1b29a73ab768ff06"} Oct 29 22:40:25.842042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.841975 5367 scope.go:122] "RemoveContainer" containerID="232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4" Oct 29 22:40:25.857633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.857613 5367 scope.go:122] "RemoveContainer" containerID="232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4" Oct 29 22:40:25.858057 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:40:25.858035 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4\": container with ID starting with 232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4 not found: ID does not exist" containerID="232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4" Oct 29 22:40:25.858167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.858063 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4"} err="failed to get container status \"232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4\": rpc error: code = NotFound desc = could not find container \"232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4\": container with ID starting with 232f1d5c0d8ed72a53365fe6ee4fb477b140d568a00994dcb99a540f797713d4 not found: ID does not exist" Oct 29 22:40:25.908738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.908683 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg"] Oct 29 22:40:25.922775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:25.922742 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-v9zrg"] Oct 29 22:40:26.756934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.756899 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz"] Oct 29 22:40:26.757761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.757745 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerName="payload-pre-processing" Oct 29 22:40:26.757818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.757762 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerName="payload-pre-processing" Oct 29 22:40:26.757948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.757939 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" containerName="payload-pre-processing" Oct 29 22:40:26.763116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.763099 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:26.781802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.781776 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz"] Oct 29 22:40:26.787956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.787926 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:40:26.788090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.787960 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:40:26.920131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.920103 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5dv\" (UniqueName: \"kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:26.920376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.920354 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:26.920506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:26.920486 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.021813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.021692 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5dv\" (UniqueName: \"kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.021999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.021979 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.022111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.022096 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.023049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.023022 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.024989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.024963 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.044462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.044422 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5dv\" (UniqueName: \"kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv\") pod \"maas-api-revoke-sub-65e0e46a-ckttz\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.084268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.084237 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:27.225196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.225161 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b8d2dd-4f61-48ee-9f67-482aea98f1d0" path="/var/lib/kubelet/pods/e1b8d2dd-4f61-48ee-9f67-482aea98f1d0/volumes" Oct 29 22:40:27.419388 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:40:27.419355 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf889cf_00cf_4e06_bf6e_13d7583de082.slice/crio-607028a1ae8c3b9eda4a3d94e39cb5ec70eb0a0bdfa130361b83ec60ca5cdb1c WatchSource:0}: Error finding container 607028a1ae8c3b9eda4a3d94e39cb5ec70eb0a0bdfa130361b83ec60ca5cdb1c: Status 404 returned error can't find the container with id 607028a1ae8c3b9eda4a3d94e39cb5ec70eb0a0bdfa130361b83ec60ca5cdb1c Oct 29 22:40:27.421013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.420985 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz"] Oct 29 22:40:27.864956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:27.864923 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" event={"ID":"8cf889cf-00cf-4e06-bf6e-13d7583de082","Type":"ContainerStarted","Data":"607028a1ae8c3b9eda4a3d94e39cb5ec70eb0a0bdfa130361b83ec60ca5cdb1c"} Oct 29 22:40:28.326242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.326209 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp"] Oct 29 22:40:28.331034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.331014 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.351296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.351267 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp"] Oct 29 22:40:28.460889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.460859 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zkl\" (UniqueName: \"kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.461098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.461040 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.461098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.461063 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.562656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.562620 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zkl\" (UniqueName: \"kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.562915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.562880 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.562915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.562910 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.563756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.563687 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.565290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.565268 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.583060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.583004 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zkl\" (UniqueName: \"kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl\") pod \"maas-api-revoke-sub-2843a034-bktmp\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.654785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.654755 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:28.892373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.892276 5367 generic.go:356] "Generic (PLEG): container finished" podID="8cf889cf-00cf-4e06-bf6e-13d7583de082" containerID="c08c2a3dbfa8a31b84b1e9b894a7696abd13694d47c3e0129e0b2e8383552983" exitCode=0 Oct 29 22:40:28.892863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:28.892516 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" event={"ID":"8cf889cf-00cf-4e06-bf6e-13d7583de082","Type":"ContainerDied","Data":"c08c2a3dbfa8a31b84b1e9b894a7696abd13694d47c3e0129e0b2e8383552983"} Oct 29 22:40:29.057284 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:40:29.057253 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa54447a_c35c_4b53_b5bf_d6936a391ae9.slice/crio-74b09974fcd7b5aeb255d1eb5ea7c2a2004f215c32247f06e541a5fe0da4d3b5 WatchSource:0}: Error finding container 74b09974fcd7b5aeb255d1eb5ea7c2a2004f215c32247f06e541a5fe0da4d3b5: Status 404 returned error can't find the container with id 74b09974fcd7b5aeb255d1eb5ea7c2a2004f215c32247f06e541a5fe0da4d3b5 Oct 29 22:40:29.057423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:29.057398 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp"] Oct 29 22:40:29.934523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:29.934482 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" event={"ID":"fa54447a-c35c-4b53-b5bf-d6936a391ae9","Type":"ContainerStarted","Data":"b4bb0ca2a093fa4cf5e5b927da4aeafc2bb5636ba5e2ea46e3e24e5f293d05fb"} Oct 29 22:40:29.936675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:29.936647 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" event={"ID":"fa54447a-c35c-4b53-b5bf-d6936a391ae9","Type":"ContainerStarted","Data":"74b09974fcd7b5aeb255d1eb5ea7c2a2004f215c32247f06e541a5fe0da4d3b5"} Oct 29 22:40:29.998767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:29.994270 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" podStartSLOduration=1.9942461439999999 podStartE2EDuration="1.994246144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:29.975729462 +0000 UTC m=+1496.869804386" watchObservedRunningTime="2026-10-29 22:40:29.994246144 +0000 UTC m=+1496.888321069" Oct 29 22:40:30.209493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.209464 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:30.230116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.230090 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv\" (UniqueName: \"kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv\") pod \"8cf889cf-00cf-4e06-bf6e-13d7583de082\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " Oct 29 22:40:30.230260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.230168 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca\") pod \"8cf889cf-00cf-4e06-bf6e-13d7583de082\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " Oct 29 22:40:30.230355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.230340 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca\") pod \"8cf889cf-00cf-4e06-bf6e-13d7583de082\" (UID: \"8cf889cf-00cf-4e06-bf6e-13d7583de082\") " Oct 29 22:40:30.230513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.230488 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca" pod "8cf889cf-00cf-4e06-bf6e-13d7583de082" (UID: "8cf889cf-00cf-4e06-bf6e-13d7583de082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:30.231239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.231218 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:30.232418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.232397 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv" pod "8cf889cf-00cf-4e06-bf6e-13d7583de082" (UID: "8cf889cf-00cf-4e06-bf6e-13d7583de082"). InnerVolumeSpecName "kube-api-access-6l5dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:30.232500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.232396 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca" pod "8cf889cf-00cf-4e06-bf6e-13d7583de082" (UID: "8cf889cf-00cf-4e06-bf6e-13d7583de082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:30.332676 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.332641 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l5dv\" (UniqueName: \"kubernetes.io/projected/8cf889cf-00cf-4e06-bf6e-13d7583de082-kube-api-access-6l5dv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:30.332676 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.332671 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf889cf-00cf-4e06-bf6e-13d7583de082-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:30.944262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.944229 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" Oct 29 22:40:30.944262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.944249 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz" event={"ID":"8cf889cf-00cf-4e06-bf6e-13d7583de082","Type":"ContainerDied","Data":"607028a1ae8c3b9eda4a3d94e39cb5ec70eb0a0bdfa130361b83ec60ca5cdb1c"} Oct 29 22:40:30.944800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.944301 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="607028a1ae8c3b9eda4a3d94e39cb5ec70eb0a0bdfa130361b83ec60ca5cdb1c" Oct 29 22:40:30.946099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.946072 5367 generic.go:356] "Generic (PLEG): container finished" podID="fa54447a-c35c-4b53-b5bf-d6936a391ae9" containerID="b4bb0ca2a093fa4cf5e5b927da4aeafc2bb5636ba5e2ea46e3e24e5f293d05fb" exitCode=0 Oct 29 22:40:30.946263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:30.946161 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" event={"ID":"fa54447a-c35c-4b53-b5bf-d6936a391ae9","Type":"ContainerDied","Data":"b4bb0ca2a093fa4cf5e5b927da4aeafc2bb5636ba5e2ea46e3e24e5f293d05fb"} Oct 29 22:40:32.150428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.150390 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:32.302331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.275352 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca\") pod \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " Oct 29 22:40:32.341799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.324210 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca" pod "fa54447a-c35c-4b53-b5bf-d6936a391ae9" (UID: "fa54447a-c35c-4b53-b5bf-d6936a391ae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:32.341799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.329157 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca\") pod \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " Oct 29 22:40:32.341799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.329269 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl\" (UniqueName: \"kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl\") pod \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\" (UID: \"fa54447a-c35c-4b53-b5bf-d6936a391ae9\") " Oct 29 22:40:32.341799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.330346 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca" pod "fa54447a-c35c-4b53-b5bf-d6936a391ae9" (UID: "fa54447a-c35c-4b53-b5bf-d6936a391ae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:32.341799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.330414 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:32.346647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.345336 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl" pod "fa54447a-c35c-4b53-b5bf-d6936a391ae9" (UID: "fa54447a-c35c-4b53-b5bf-d6936a391ae9"). InnerVolumeSpecName "kube-api-access-29zkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:32.432480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.432441 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zkl\" (UniqueName: \"kubernetes.io/projected/fa54447a-c35c-4b53-b5bf-d6936a391ae9-kube-api-access-29zkl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:32.432480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.432472 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa54447a-c35c-4b53-b5bf-d6936a391ae9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:32.966868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.966841 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" Oct 29 22:40:32.967048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.966882 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp" event={"ID":"fa54447a-c35c-4b53-b5bf-d6936a391ae9","Type":"ContainerDied","Data":"74b09974fcd7b5aeb255d1eb5ea7c2a2004f215c32247f06e541a5fe0da4d3b5"} Oct 29 22:40:32.967048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:32.966953 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74b09974fcd7b5aeb255d1eb5ea7c2a2004f215c32247f06e541a5fe0da4d3b5" Oct 29 22:40:35.994454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:35.994424 5367 scope.go:122] "RemoveContainer" containerID="e1a255e5aa4d4853a7604eea2759a130d504ccc2fadb2fb106e80c1adfd70c4a" Oct 29 22:40:36.010630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.010609 5367 scope.go:122] "RemoveContainer" containerID="589979fee162eb7dd4e30090897a9aabcf61dbffecd672ae9780b543aec3ab6a" Oct 29 22:40:36.021087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.021064 5367 scope.go:122] "RemoveContainer" containerID="79f826af8055c367db0b0a620e23856e6c8ed7eff6f94b6c79ad0210f627e149" Oct 29 22:40:36.034379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.034359 5367 scope.go:122] "RemoveContainer" containerID="c681bd642c66e9441817abff5ffbb6bff15703d0300ac593cdb24a40985db449" Oct 29 22:40:36.046931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.046902 5367 scope.go:122] "RemoveContainer" containerID="024414ec47bf3f3eaa395bf4da874c4b28c929121df5afad8cb6e3d338be93ca" Oct 29 22:40:36.057966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.057947 5367 scope.go:122] "RemoveContainer" containerID="1b370dbe39f2cf2a21584f9a6b27c7ae604d5b63d7641232003d8e2698c974e1" Oct 29 22:40:36.068018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.067999 5367 scope.go:122] "RemoveContainer" containerID="51fd779e50b5ce36ce377ee427d2c398aaf44e6cc7b02b10596cfa4e992abdcf" Oct 29 22:40:36.078971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.078955 5367 scope.go:122] "RemoveContainer" containerID="d47fb550474c4b3d7b1bde746bc2fe0d555649f96f1bcd214cd21b78a507519f" Oct 29 22:40:36.088672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.088655 5367 scope.go:122] "RemoveContainer" containerID="93dd4dbdd62dc89b483371981d761e0306c875b6fe726af8a9e35ac9ccc9c9f1" Oct 29 22:40:36.098775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:36.098757 5367 scope.go:122] "RemoveContainer" containerID="a0b34498931a705d97262ab653e68a3766a8e44dde76b719609a0d18a29bb74d" Oct 29 22:40:50.584230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:50.584192 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl"] Oct 29 22:40:50.584633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:50.584402 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" containerID="cri-o://a8ef8bd1ce4900a1907a1b438b06625cc686f4064c84d4c1e81ccfdaabc3c8c9" gracePeriod=30 Oct 29 22:40:52.334959 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:40:52.334918 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:40:56.553724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.526971 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:40:56.622779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.620760 5367 generic.go:356] "Generic (PLEG): container finished" podID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerID="a8ef8bd1ce4900a1907a1b438b06625cc686f4064c84d4c1e81ccfdaabc3c8c9" exitCode=0 Oct 29 22:40:56.622779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.620931 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" event={"ID":"9be98a21-c1ba-4ecf-9678-330ec53a8106","Type":"ContainerDied","Data":"a8ef8bd1ce4900a1907a1b438b06625cc686f4064c84d4c1e81ccfdaabc3c8c9"} Oct 29 22:40:56.622779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.620993 5367 scope.go:122] "RemoveContainer" containerID="a8ef8bd1ce4900a1907a1b438b06625cc686f4064c84d4c1e81ccfdaabc3c8c9" Oct 29 22:40:56.645931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.644897 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" Oct 29 22:40:56.698438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.698381 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.698776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.698751 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.699046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.699028 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.699278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.699261 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.699440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.699424 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.699632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.699616 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.699950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.699932 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.700103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.700088 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.725345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.725219 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:56.750557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.700257 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token\") pod \"9be98a21-c1ba-4ecf-9678-330ec53a8106\" (UID: \"9be98a21-c1ba-4ecf-9678-330ec53a8106\") " Oct 29 22:40:56.761321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.760992 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:56.790728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.777322 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:56.790728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.777372 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:56.790728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.777737 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:56.790728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.778081 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:57.001733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:56.997810 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:57.025728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.024251 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:40:57.118998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.118944 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "kube-api-access-2tnk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:57.125198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.119285 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:57.176025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.175350 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:57.176225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.175927 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token" pod "9be98a21-c1ba-4ecf-9678-330ec53a8106" (UID: "9be98a21-c1ba-4ecf-9678-330ec53a8106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.226851 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.240895 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248302 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248332 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248348 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be98a21-c1ba-4ecf-9678-330ec53a8106-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248361 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248380 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248394 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tnk6\" (UniqueName: \"kubernetes.io/projected/9be98a21-c1ba-4ecf-9678-330ec53a8106-kube-api-access-2tnk6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.266748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.248410 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be98a21-c1ba-4ecf-9678-330ec53a8106-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.331975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.331926 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.462782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.371775 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:40:57.487723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.390383 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.487723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.410383 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.487723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.410471 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.599722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.599334 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.785778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.785098 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:40:57.809724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.787777 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.809724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.788159 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:57.827943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.826198 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4"] Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865443 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8cf889cf-00cf-4e06-bf6e-13d7583de082" containerName="revoke-keys" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865473 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf889cf-00cf-4e06-bf6e-13d7583de082" containerName="revoke-keys" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865503 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865511 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865532 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fa54447a-c35c-4b53-b5bf-d6936a391ae9" containerName="revoke-keys" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865540 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa54447a-c35c-4b53-b5bf-d6936a391ae9" containerName="revoke-keys" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865797 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf889cf-00cf-4e06-bf6e-13d7583de082" containerName="revoke-keys" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865815 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" containerName="istio-proxy" Oct 29 22:40:57.874726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.865828 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa54447a-c35c-4b53-b5bf-d6936a391ae9" containerName="revoke-keys" Oct 29 22:40:57.908957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.908915 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4"] Oct 29 22:40:57.909280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.909259 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:57.922278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.914295 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl" event={"ID":"9be98a21-c1ba-4ecf-9678-330ec53a8106","Type":"ContainerDied","Data":"09ee2a2f2e4add1739e6fdeaba283ca0fc9d37ed7c6b442fec7fc8625e3bfb8c"} Oct 29 22:40:57.951100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:57.949295 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e224cd-serving-cert\"" Oct 29 22:40:58.038776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.038590 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:40:58.060793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.056912 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:40:58.087228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.087190 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.096965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.096932 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnvz\" (UniqueName: \"kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.097433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.097396 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.097644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.097615 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.146829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.146753 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl"] Oct 29 22:40:58.206836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.206788 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:58.212720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.208421 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.240867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.240827 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.241474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.241444 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnvz\" (UniqueName: \"kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.241869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.241846 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.263252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.263221 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.268738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.265022 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:58.268738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.266525 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:40:58.268738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.267499 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:40:58.268738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.267557 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:40:58.306884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.306753 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-4vrbl"] Oct 29 22:40:58.369729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.343788 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.397724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.369970 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:58.397724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.370367 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:40:58.397724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.386763 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnvz\" (UniqueName: \"kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.556772 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.557070 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.557306 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.557515 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.558173 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.558229 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.558264 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:40:58.559775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.558296 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:40:58.601786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.579796 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs\") pod \"maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.604594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.602405 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:58.604594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.603012 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:40:58.716723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.671068 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:40:58.721524 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.721407 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:40:58.724005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.723785 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:40:58.794845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.794798 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.800271 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.797302 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.819473 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.821793 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.821830 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.821867 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:40:58.851771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.821899 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:40:58.999749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:58.988316 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:40:59.310290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.282090 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:40:59.340570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.340533 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be98a21-c1ba-4ecf-9678-330ec53a8106" path="/var/lib/kubelet/pods/9be98a21-c1ba-4ecf-9678-330ec53a8106/volumes" Oct 29 22:40:59.382849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.381644 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:40:59.445165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.443538 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:40:59.449896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.449050 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:40:59.551720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.546364 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:59.598215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.566006 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:59.598215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.566980 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:40:59.639219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.639189 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:40:59.645461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.645432 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:40:59.751082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.743248 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:40:59.837985 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:40:59.837951 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e1e05f_1de1_46fd_b66f_d376d29692b1.slice/crio-308cf2126c10f45d25537818c218ab1f5518f4109d43a59c3256c8ca8cfddf7b WatchSource:0}: Error finding container 308cf2126c10f45d25537818c218ab1f5518f4109d43a59c3256c8ca8cfddf7b: Status 404 returned error can't find the container with id 308cf2126c10f45d25537818c218ab1f5518f4109d43a59c3256c8ca8cfddf7b Oct 29 22:40:59.841639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.840986 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:40:59.940610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:40:59.940576 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4"] Oct 29 22:41:00.224408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:00.224325 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" event={"ID":"58e1e05f-1de1-46fd-b66f-d376d29692b1","Type":"ContainerStarted","Data":"308cf2126c10f45d25537818c218ab1f5518f4109d43a59c3256c8ca8cfddf7b"} Oct 29 22:41:01.249229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:01.249195 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" event={"ID":"58e1e05f-1de1-46fd-b66f-d376d29692b1","Type":"ContainerStarted","Data":"aad58c03db146a96830c3ca394339382b560175ff8d3b3d1503397725d79c637"} Oct 29 22:41:01.251915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:01.249275 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:41:01.325028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:01.324974 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" podStartSLOduration=14.925369467 podStartE2EDuration="15.324958582s" totalImagesPullingTime="399.589115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:01.305058042 +0000 UTC m=+1528.199132965" watchObservedRunningTime="2026-10-29 22:41:01.324958582 +0000 UTC m=+1528.219033527" Oct 29 22:41:06.031981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:06.031945 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd"] Oct 29 22:41:06.042458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:06.042423 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac561c2c-qztxd"] Oct 29 22:41:07.224751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:07.224718 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f825084-5ce2-447d-ace5-80c875e3b95b" path="/var/lib/kubelet/pods/1f825084-5ce2-447d-ace5-80c875e3b95b/volumes" Oct 29 22:41:07.278093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:07.278066 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:41:08.346269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.346237 5367 generic.go:356] "Generic (PLEG): container finished" podID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerID="793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a" exitCode=0 Oct 29 22:41:08.346857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.346289 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" event={"ID":"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3","Type":"ContainerDied","Data":"793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a"} Oct 29 22:41:08.346929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.346889 5367 scope.go:122] "RemoveContainer" containerID="793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a" Oct 29 22:41:08.348401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.348379 5367 generic.go:356] "Generic (PLEG): container finished" podID="6be234cd-119e-42c7-918d-03b3259114b4" containerID="7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c" exitCode=0 Oct 29 22:41:08.348499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.348434 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" event={"ID":"6be234cd-119e-42c7-918d-03b3259114b4","Type":"ContainerDied","Data":"7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c"} Oct 29 22:41:08.348909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.348892 5367 scope.go:122] "RemoveContainer" containerID="7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c" Oct 29 22:41:08.349165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.349136 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:08.349247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.349228 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:08.349310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.349282 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:08.350621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.350594 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:08.350680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.350664 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:08.350745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:08.350732 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:09.404833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:09.404748 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" event={"ID":"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3","Type":"ContainerStarted","Data":"ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc"} Oct 29 22:41:09.410116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:09.410018 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" event={"ID":"6be234cd-119e-42c7-918d-03b3259114b4","Type":"ContainerStarted","Data":"9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970"} Oct 29 22:41:09.449651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:09.449624 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:41:09.452863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:09.452836 5367 patch_prober.go:29] interesting pod/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 29 22:41:09.452996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:09.452906 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 29 22:41:09.997722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:09.993939 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:41:10.008731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:10.005995 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 29 22:41:10.008731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:10.006087 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 29 22:41:10.450226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:10.450199 5367 patch_prober.go:29] interesting pod/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 29 22:41:10.450731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:10.450686 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 29 22:41:11.009724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:11.009333 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.009724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:11.009396 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 29 22:41:11.451365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:11.451323 5367 patch_prober.go:29] interesting pod/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.451943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:11.451397 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 29 22:41:11.993825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:11.993789 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.994117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:11.994085 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 29 22:41:12.450031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:12.449996 5367 patch_prober.go:29] interesting pod/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 29 22:41:12.450209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:12.450061 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 29 22:41:12.994722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:12.994439 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 29 22:41:12.994722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:12.994532 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 29 22:41:13.107482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:13.107424 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:41:13.449559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:13.449525 5367 patch_prober.go:29] interesting pod/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 29 22:41:13.449785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:13.449582 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 29 22:41:14.003186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.003144 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 29 22:41:14.003833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.003798 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 29 22:41:14.117001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.116962 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:41:14.449373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.449338 5367 patch_prober.go:29] interesting pod/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 29 22:41:14.449712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.449672 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 29 22:41:14.993089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.993057 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 29 22:41:14.993276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:14.993112 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 29 22:41:15.993625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:15.993589 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:41:15.995609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:15.995589 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:41:15.998860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:15.998841 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:41:15.999954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:15.999936 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:41:18.148846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.148810 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx"] Oct 29 22:41:18.152786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.152767 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.173281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.173252 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx"] Oct 29 22:41:18.180598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.180264 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:41:18.180598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.180395 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:41:18.282894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.282852 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.283086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.283020 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.283136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.283087 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2h7\" (UniqueName: \"kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.385081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.385049 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.385277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.385162 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.385277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.385202 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2h7\" (UniqueName: \"kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.385895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.385872 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.387562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.387537 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.405767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.405684 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2h7\" (UniqueName: \"kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7\") pod \"maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.467264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.467229 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:18.776649 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:41:18.776613 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1e270e_7912_4594_9126_dfd633567075.slice/crio-292f5e28b4395bd5c04e3535e80c92d8ed2716cefcb72946e3eb8719d8e77a98 WatchSource:0}: Error finding container 292f5e28b4395bd5c04e3535e80c92d8ed2716cefcb72946e3eb8719d8e77a98: Status 404 returned error can't find the container with id 292f5e28b4395bd5c04e3535e80c92d8ed2716cefcb72946e3eb8719d8e77a98 Oct 29 22:41:18.777162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:18.777107 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx"] Oct 29 22:41:19.305437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.305401 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8"] Oct 29 22:41:19.309417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.309400 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.328585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.328553 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8"] Oct 29 22:41:19.401662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.401619 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.401890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.401865 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.401971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.401907 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l57n\" (UniqueName: \"kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.503835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.503741 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.504004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.503847 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.504004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.503866 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l57n\" (UniqueName: \"kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.504480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.504456 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.506223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.506200 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.524378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.524349 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l57n\" (UniqueName: \"kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n\") pod \"maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.624574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.624537 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:19.666857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.666815 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" event={"ID":"5a1e270e-7912-4594-9126-dfd633567075","Type":"ContainerStarted","Data":"0cb4c997496adfd4ddfea5af874426db50dbacfe236efbaebb96588a2fef313e"} Oct 29 22:41:19.667051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.666883 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" event={"ID":"5a1e270e-7912-4594-9126-dfd633567075","Type":"ContainerStarted","Data":"292f5e28b4395bd5c04e3535e80c92d8ed2716cefcb72946e3eb8719d8e77a98"} Oct 29 22:41:19.713834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.712637 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" podStartSLOduration=1.7126182810000001 podStartE2EDuration="1.712618281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:19.706181405 +0000 UTC m=+1546.600256326" watchObservedRunningTime="2026-10-29 22:41:19.712618281 +0000 UTC m=+1546.606693199" Oct 29 22:41:19.935482 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:41:19.935441 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32025844_1d97_468f_b79d_638da9e02240.slice/crio-625b31df26c00e68a5b5b01f258340dd7daa7dbd9d020d873b5b55c6e2809b75 WatchSource:0}: Error finding container 625b31df26c00e68a5b5b01f258340dd7daa7dbd9d020d873b5b55c6e2809b75: Status 404 returned error can't find the container with id 625b31df26c00e68a5b5b01f258340dd7daa7dbd9d020d873b5b55c6e2809b75 Oct 29 22:41:19.936089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:19.935991 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8"] Oct 29 22:41:20.678416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:20.678377 5367 generic.go:356] "Generic (PLEG): container finished" podID="5a1e270e-7912-4594-9126-dfd633567075" containerID="0cb4c997496adfd4ddfea5af874426db50dbacfe236efbaebb96588a2fef313e" exitCode=0 Oct 29 22:41:20.678931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:20.678477 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" event={"ID":"5a1e270e-7912-4594-9126-dfd633567075","Type":"ContainerDied","Data":"0cb4c997496adfd4ddfea5af874426db50dbacfe236efbaebb96588a2fef313e"} Oct 29 22:41:20.680230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:20.680208 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" event={"ID":"32025844-1d97-468f-b79d-638da9e02240","Type":"ContainerStarted","Data":"70ca8fa742bbff30d0ef6d5ba38dd8dd19384a4dc07408369ef2afe6cadcb98f"} Oct 29 22:41:20.680340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:20.680246 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" event={"ID":"32025844-1d97-468f-b79d-638da9e02240","Type":"ContainerStarted","Data":"625b31df26c00e68a5b5b01f258340dd7daa7dbd9d020d873b5b55c6e2809b75"} Oct 29 22:41:20.763821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:20.763768 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" podStartSLOduration=1.763753259 podStartE2EDuration="1.763753259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:20.758526647 +0000 UTC m=+1547.652601571" watchObservedRunningTime="2026-10-29 22:41:20.763753259 +0000 UTC m=+1547.657828179" Oct 29 22:41:21.691174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:21.691140 5367 generic.go:356] "Generic (PLEG): container finished" podID="32025844-1d97-468f-b79d-638da9e02240" containerID="70ca8fa742bbff30d0ef6d5ba38dd8dd19384a4dc07408369ef2afe6cadcb98f" exitCode=0 Oct 29 22:41:21.691611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:21.691213 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" event={"ID":"32025844-1d97-468f-b79d-638da9e02240","Type":"ContainerDied","Data":"70ca8fa742bbff30d0ef6d5ba38dd8dd19384a4dc07408369ef2afe6cadcb98f"} Oct 29 22:41:21.969764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:21.969733 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:22.154279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.154246 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca\") pod \"5a1e270e-7912-4594-9126-dfd633567075\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " Oct 29 22:41:22.154279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.154285 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7\" (UniqueName: \"kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7\") pod \"5a1e270e-7912-4594-9126-dfd633567075\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " Oct 29 22:41:22.154598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.154346 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca\") pod \"5a1e270e-7912-4594-9126-dfd633567075\" (UID: \"5a1e270e-7912-4594-9126-dfd633567075\") " Oct 29 22:41:22.154799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.154773 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca" pod "5a1e270e-7912-4594-9126-dfd633567075" (UID: "5a1e270e-7912-4594-9126-dfd633567075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:22.155178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.155157 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e270e-7912-4594-9126-dfd633567075-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:22.156593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.156570 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca" pod "5a1e270e-7912-4594-9126-dfd633567075" (UID: "5a1e270e-7912-4594-9126-dfd633567075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:22.156746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.156687 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7" pod "5a1e270e-7912-4594-9126-dfd633567075" (UID: "5a1e270e-7912-4594-9126-dfd633567075"). InnerVolumeSpecName "kube-api-access-gp2h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:22.257157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.257064 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp2h7\" (UniqueName: \"kubernetes.io/projected/5a1e270e-7912-4594-9126-dfd633567075-kube-api-access-gp2h7\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:22.257157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.257098 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e270e-7912-4594-9126-dfd633567075-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:22.333785 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:41:22.333751 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:41:22.701130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.701102 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" Oct 29 22:41:22.701548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.701134 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx" event={"ID":"5a1e270e-7912-4594-9126-dfd633567075","Type":"ContainerDied","Data":"292f5e28b4395bd5c04e3535e80c92d8ed2716cefcb72946e3eb8719d8e77a98"} Oct 29 22:41:22.701548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.701188 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="292f5e28b4395bd5c04e3535e80c92d8ed2716cefcb72946e3eb8719d8e77a98" Oct 29 22:41:22.885530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:22.885500 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:23.071229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.071186 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca\") pod \"32025844-1d97-468f-b79d-638da9e02240\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " Oct 29 22:41:23.071414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.071242 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n\" (UniqueName: \"kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n\") pod \"32025844-1d97-468f-b79d-638da9e02240\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " Oct 29 22:41:23.071414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.071355 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca\") pod \"32025844-1d97-468f-b79d-638da9e02240\" (UID: \"32025844-1d97-468f-b79d-638da9e02240\") " Oct 29 22:41:23.071860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.071831 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca" pod "32025844-1d97-468f-b79d-638da9e02240" (UID: "32025844-1d97-468f-b79d-638da9e02240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:23.072148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.072131 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32025844-1d97-468f-b79d-638da9e02240-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:23.073987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.073960 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca" pod "32025844-1d97-468f-b79d-638da9e02240" (UID: "32025844-1d97-468f-b79d-638da9e02240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:23.074079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.074005 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n" pod "32025844-1d97-468f-b79d-638da9e02240" (UID: "32025844-1d97-468f-b79d-638da9e02240"). InnerVolumeSpecName "kube-api-access-5l57n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:23.173390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.173353 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32025844-1d97-468f-b79d-638da9e02240-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:23.173390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.173385 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l57n\" (UniqueName: \"kubernetes.io/projected/32025844-1d97-468f-b79d-638da9e02240-kube-api-access-5l57n\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:23.711225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.711192 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" Oct 29 22:41:23.711630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.711198 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8" event={"ID":"32025844-1d97-468f-b79d-638da9e02240","Type":"ContainerDied","Data":"625b31df26c00e68a5b5b01f258340dd7daa7dbd9d020d873b5b55c6e2809b75"} Oct 29 22:41:23.711630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:23.711316 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625b31df26c00e68a5b5b01f258340dd7daa7dbd9d020d873b5b55c6e2809b75" Oct 29 22:41:24.996560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.996523 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc"] Oct 29 22:41:24.997368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.997353 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5a1e270e-7912-4594-9126-dfd633567075" containerName="revoke-keys" Oct 29 22:41:24.997427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.997369 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1e270e-7912-4594-9126-dfd633567075" containerName="revoke-keys" Oct 29 22:41:24.997427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.997381 5367 cpu_manager.go:414] "containerMap: removing container" podUID="32025844-1d97-468f-b79d-638da9e02240" containerName="revoke-keys" Oct 29 22:41:24.997427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.997386 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32025844-1d97-468f-b79d-638da9e02240" containerName="revoke-keys" Oct 29 22:41:24.997547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.997538 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1e270e-7912-4594-9126-dfd633567075" containerName="revoke-keys" Oct 29 22:41:24.997587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:24.997549 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="32025844-1d97-468f-b79d-638da9e02240" containerName="revoke-keys" Oct 29 22:41:25.000485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.000466 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.017958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.017913 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d478b36e\"" Oct 29 22:41:25.118808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.078119 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d478b36e-dockercfg-g5dhv\"" Oct 29 22:41:25.179822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.161826 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc"] Oct 29 22:41:25.235721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.219865 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd592\" (UniqueName: \"kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.235721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.220051 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.333723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.325878 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd592\" (UniqueName: \"kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.333723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.326090 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.333723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.327404 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.400106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.400073 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd592\" (UniqueName: \"kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592\") pod \"payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.614581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.614486 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:25.922324 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:41:25.922291 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4520c6_0cc3_4c4f_b387_0d63d9f757fc.slice/crio-2ff9d0e1ddbc5a1a24be3b58d746253d5208af1982fc6bc2b03c33586fc4a5e0 WatchSource:0}: Error finding container 2ff9d0e1ddbc5a1a24be3b58d746253d5208af1982fc6bc2b03c33586fc4a5e0: Status 404 returned error can't find the container with id 2ff9d0e1ddbc5a1a24be3b58d746253d5208af1982fc6bc2b03c33586fc4a5e0 Oct 29 22:41:25.922564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:25.922540 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc"] Oct 29 22:41:26.741499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:26.741465 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" event={"ID":"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc","Type":"ContainerStarted","Data":"b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e"} Oct 29 22:41:26.741984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:26.741519 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" event={"ID":"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc","Type":"ContainerStarted","Data":"2ff9d0e1ddbc5a1a24be3b58d746253d5208af1982fc6bc2b03c33586fc4a5e0"} Oct 29 22:41:26.741984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:26.741685 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:26.788801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:26.788736 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" podStartSLOduration=2.788716171 podStartE2EDuration="2.788716171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:26.783067471 +0000 UTC m=+1553.677142419" watchObservedRunningTime="2026-10-29 22:41:26.788716171 +0000 UTC m=+1553.682791083" Oct 29 22:41:28.591911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:28.591878 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9"] Oct 29 22:41:29.863222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:29.863188 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd"] Oct 29 22:41:29.898009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:29.897972 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728"] Oct 29 22:41:33.598724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:33.598591 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" podUID="e2c191ad-46e4-451e-afe2-ce43bead4d30" containerName="payload-pre-processing" containerID="cri-o://7b16186dc7a7a4946e654d94ddd8458260de1f20da0e3a8dc232b705951272f2" gracePeriod=25 Oct 29 22:41:33.823750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:33.823715 5367 generic.go:356] "Generic (PLEG): container finished" podID="e2c191ad-46e4-451e-afe2-ce43bead4d30" containerID="7b16186dc7a7a4946e654d94ddd8458260de1f20da0e3a8dc232b705951272f2" exitCode=0 Oct 29 22:41:33.824047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:33.824032 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" event={"ID":"e2c191ad-46e4-451e-afe2-ce43bead4d30","Type":"ContainerDied","Data":"7b16186dc7a7a4946e654d94ddd8458260de1f20da0e3a8dc232b705951272f2"} Oct 29 22:41:34.035846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.035811 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:41:34.100373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.100274 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume\") pod \"e2c191ad-46e4-451e-afe2-ce43bead4d30\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " Oct 29 22:41:34.100373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.100334 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j\" (UniqueName: \"kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j\") pod \"e2c191ad-46e4-451e-afe2-ce43bead4d30\" (UID: \"e2c191ad-46e4-451e-afe2-ce43bead4d30\") " Oct 29 22:41:34.101180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.101146 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume" pod "e2c191ad-46e4-451e-afe2-ce43bead4d30" (UID: "e2c191ad-46e4-451e-afe2-ce43bead4d30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:34.101463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.101434 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c191ad-46e4-451e-afe2-ce43bead4d30-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:34.103495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.103466 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j" pod "e2c191ad-46e4-451e-afe2-ce43bead4d30" (UID: "e2c191ad-46e4-451e-afe2-ce43bead4d30"). InnerVolumeSpecName "kube-api-access-g7n6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:34.203561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.203518 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7n6j\" (UniqueName: \"kubernetes.io/projected/e2c191ad-46e4-451e-afe2-ce43bead4d30-kube-api-access-g7n6j\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:34.835714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.835669 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" Oct 29 22:41:34.836251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.835721 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9" event={"ID":"e2c191ad-46e4-451e-afe2-ce43bead4d30","Type":"ContainerDied","Data":"9b8b4c45b676c6984f797772c328b0837d114421d3e39cae47ab47bac1f20123"} Oct 29 22:41:34.836251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.835784 5367 scope.go:122] "RemoveContainer" containerID="7b16186dc7a7a4946e654d94ddd8458260de1f20da0e3a8dc232b705951272f2" Oct 29 22:41:34.864576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.864522 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" podUID="5d70bc4f-1677-4043-9da2-fe872c4a92e6" containerName="payload-processing" containerID="cri-o://b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d" gracePeriod=25 Oct 29 22:41:34.893322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.893290 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9"] Oct 29 22:41:34.899117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.899035 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" podUID="f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" containerName="payload-pre-processing" containerID="cri-o://22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948" gracePeriod=25 Oct 29 22:41:34.906761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:34.906732 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0ab33e-5f5d8f6459-frxg9"] Oct 29 22:41:35.196148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.196114 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:41:35.226527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.226493 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c191ad-46e4-451e-afe2-ce43bead4d30" path="/var/lib/kubelet/pods/e2c191ad-46e4-451e-afe2-ce43bead4d30/volumes" Oct 29 22:41:35.242637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.242609 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:41:35.297184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.297151 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85"] Oct 29 22:41:35.298266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298239 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5d70bc4f-1677-4043-9da2-fe872c4a92e6" containerName="payload-processing" Oct 29 22:41:35.298266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298267 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d70bc4f-1677-4043-9da2-fe872c4a92e6" containerName="payload-processing" Oct 29 22:41:35.298434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298283 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e2c191ad-46e4-451e-afe2-ce43bead4d30" containerName="payload-pre-processing" Oct 29 22:41:35.298434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298291 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c191ad-46e4-451e-afe2-ce43bead4d30" containerName="payload-pre-processing" Oct 29 22:41:35.298434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298310 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" containerName="payload-pre-processing" Oct 29 22:41:35.298434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298319 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" containerName="payload-pre-processing" Oct 29 22:41:35.298666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298648 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d70bc4f-1677-4043-9da2-fe872c4a92e6" containerName="payload-processing" Oct 29 22:41:35.298711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298674 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c191ad-46e4-451e-afe2-ce43bead4d30" containerName="payload-pre-processing" Oct 29 22:41:35.298711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.298689 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" containerName="payload-pre-processing" Oct 29 22:41:35.302355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.302334 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.345468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.345353 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-63db1a-openshift-default-dockercfg-x2525\"" Oct 29 22:41:35.366311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.366163 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm\" (UniqueName: \"kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm\") pod \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " Oct 29 22:41:35.366311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.366246 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc\" (UniqueName: \"kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc\") pod \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " Oct 29 22:41:35.366571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.366472 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume\") pod \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\" (UID: \"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8\") " Oct 29 22:41:35.366629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.366579 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume\") pod \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\" (UID: \"5d70bc4f-1677-4043-9da2-fe872c4a92e6\") " Oct 29 22:41:35.379209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.369277 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume" pod "5d70bc4f-1677-4043-9da2-fe872c4a92e6" (UID: "5d70bc4f-1677-4043-9da2-fe872c4a92e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:35.379209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.369799 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume" pod "f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" (UID: "f147bf4c-527f-4cf7-85b1-58fa6b2eedb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:35.379209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.374784 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm" pod "5d70bc4f-1677-4043-9da2-fe872c4a92e6" (UID: "5d70bc4f-1677-4043-9da2-fe872c4a92e6"). InnerVolumeSpecName "kube-api-access-l8ssm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:35.379209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.376747 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85"] Oct 29 22:41:35.379209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.378430 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc" pod "f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" (UID: "f147bf4c-527f-4cf7-85b1-58fa6b2eedb8"). InnerVolumeSpecName "kube-api-access-dgmwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:35.469227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469186 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469314 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469351 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469384 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469517 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz27p\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469759 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469788 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469871 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.469962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.469951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.470152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.470141 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d70bc4f-1677-4043-9da2-fe872c4a92e6-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:35.470185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.470153 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ssm\" (UniqueName: \"kubernetes.io/projected/5d70bc4f-1677-4043-9da2-fe872c4a92e6-kube-api-access-l8ssm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:35.470185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.470163 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmwc\" (UniqueName: \"kubernetes.io/projected/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-kube-api-access-dgmwc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:35.470185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.470172 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:41:35.571236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571197 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571281 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571384 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571437 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571501 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz27p\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571576 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571604 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571690 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571828 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.571971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.571949 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.572068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.572044 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.572285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.572258 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.572366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.572280 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.572412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.572366 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.574319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.574295 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.574401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.574392 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.600354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.600319 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.601527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.601503 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz27p\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.688275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.688187 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:35.848892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.848850 5367 generic.go:356] "Generic (PLEG): container finished" podID="5d70bc4f-1677-4043-9da2-fe872c4a92e6" containerID="b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d" exitCode=0 Oct 29 22:41:35.849380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.848984 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" Oct 29 22:41:35.849380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.849035 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" event={"ID":"5d70bc4f-1677-4043-9da2-fe872c4a92e6","Type":"ContainerDied","Data":"b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d"} Oct 29 22:41:35.849380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.849117 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd" event={"ID":"5d70bc4f-1677-4043-9da2-fe872c4a92e6","Type":"ContainerDied","Data":"17649a96140fbc2f581a91f6682d5c769041534041e5e1ad1499103c80f70a52"} Oct 29 22:41:35.849380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.849146 5367 scope.go:122] "RemoveContainer" containerID="b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d" Oct 29 22:41:35.861311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.861278 5367 generic.go:356] "Generic (PLEG): container finished" podID="f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" containerID="22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948" exitCode=0 Oct 29 22:41:35.861499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.861469 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" event={"ID":"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8","Type":"ContainerDied","Data":"22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948"} Oct 29 22:41:35.861638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.861528 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" Oct 29 22:41:35.861638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.861545 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728" event={"ID":"f147bf4c-527f-4cf7-85b1-58fa6b2eedb8","Type":"ContainerDied","Data":"d10861d696697d5aa41555d6f89c1a8b40b0eb6fe1e0ae76cf5f821f3662cbfc"} Oct 29 22:41:35.872337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.872317 5367 scope.go:122] "RemoveContainer" containerID="b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d" Oct 29 22:41:35.872648 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:41:35.872620 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d\": container with ID starting with b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d not found: ID does not exist" containerID="b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d" Oct 29 22:41:35.872727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.872660 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d"} err="failed to get container status \"b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d\": rpc error: code = NotFound desc = could not find container \"b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d\": container with ID starting with b1da12ad7575058b1c07cfcd40e419f4b5faf57d1994c88e5f6d826df9f6194d not found: ID does not exist" Oct 29 22:41:35.872727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.872684 5367 scope.go:122] "RemoveContainer" containerID="22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948" Oct 29 22:41:35.902907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.902887 5367 scope.go:122] "RemoveContainer" containerID="22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948" Oct 29 22:41:35.903272 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:41:35.903243 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948\": container with ID starting with 22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948 not found: ID does not exist" containerID="22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948" Oct 29 22:41:35.903327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.903281 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948"} err="failed to get container status \"22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948\": rpc error: code = NotFound desc = could not find container \"22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948\": container with ID starting with 22d587f128b028a53838b91341cb87b9b10863dcdf8c5316df6b73e245f54948 not found: ID does not exist" Oct 29 22:41:35.922783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.922749 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd"] Oct 29 22:41:35.928212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.928183 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76fb5-5d75b85b9d-64hcd"] Oct 29 22:41:35.964384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.964293 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728"] Oct 29 22:41:35.972113 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:35.972081 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76fb5-86dc79f6c9-6h728"] Oct 29 22:41:36.044977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.042681 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85"] Oct 29 22:41:36.053032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.052996 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:36.053142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.053071 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:36.053142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.053103 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:36.223681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.223563 5367 scope.go:122] "RemoveContainer" containerID="8c0d1678c92e54226090e7cf1f6720c658b71b5444047980a5a8d4059edc230e" Oct 29 22:41:36.245522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.245497 5367 scope.go:122] "RemoveContainer" containerID="ec9e2e1d09eb95bee9634abc7d3fd6b3dd7416a1a9510132d0fee3fee149f339" Oct 29 22:41:36.257951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.257924 5367 scope.go:122] "RemoveContainer" containerID="7be1d2c3a10c1cf170627379c4041cf6f9ffa192f0cc4c0900bb8e14fb31bbc4" Oct 29 22:41:36.872962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.872926 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" event={"ID":"5ee79d16-2b7e-4129-8cff-8235bde26d3b","Type":"ContainerStarted","Data":"51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21"} Oct 29 22:41:36.873372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.872984 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" event={"ID":"5ee79d16-2b7e-4129-8cff-8235bde26d3b","Type":"ContainerStarted","Data":"a18b9ecc0cca93928aa52362601e6f8abb7e17486a6b43c889d761e373724f7e"} Oct 29 22:41:36.920164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:36.920113 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podStartSLOduration=3.920098016 podStartE2EDuration="3.920098016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:36.917004746 +0000 UTC m=+1563.811079665" watchObservedRunningTime="2026-10-29 22:41:36.920098016 +0000 UTC m=+1563.814172935" Oct 29 22:41:37.225891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.225814 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d70bc4f-1677-4043-9da2-fe872c4a92e6" path="/var/lib/kubelet/pods/5d70bc4f-1677-4043-9da2-fe872c4a92e6/volumes" Oct 29 22:41:37.226346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.226331 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f147bf4c-527f-4cf7-85b1-58fa6b2eedb8" path="/var/lib/kubelet/pods/f147bf4c-527f-4cf7-85b1-58fa6b2eedb8/volumes" Oct 29 22:41:37.689038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.689006 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:37.696004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.695972 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:37.757054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.757026 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:41:37.882760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.882727 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:37.883717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:41:37.883683 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:41:52.336126 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:41:52.336088 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:42:22.339036 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:42:22.338653 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:42:36.351172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:36.351147 5367 scope.go:122] "RemoveContainer" containerID="2b3b1d82f8500ae2de78becfcb4ad45b33add40a317a5334b858bfbb99b3ca32" Oct 29 22:42:36.422595 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:36.422445 5367 scope.go:122] "RemoveContainer" containerID="14d3363a45d6f018aeddf8f2b37445f6ef9f14f06c8a3d9baba50b90fc7989ab" Oct 29 22:42:36.438145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:36.438114 5367 scope.go:122] "RemoveContainer" containerID="b5f682878d526cd0a92551a9861295be20fac6306bd6beda3ae88f5bbc554172" Oct 29 22:42:39.768434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:39.768403 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx"] Oct 29 22:42:39.785075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:39.785036 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0ab33e-n9xnx"] Oct 29 22:42:41.228326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:41.228294 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1e270e-7912-4594-9126-dfd633567075" path="/var/lib/kubelet/pods/5a1e270e-7912-4594-9126-dfd633567075/volumes" Oct 29 22:42:45.515575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:45.515485 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8"] Oct 29 22:42:45.530552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:45.530517 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f76fb5-8gpr8"] Oct 29 22:42:47.228166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:47.228127 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32025844-1d97-468f-b79d-638da9e02240" path="/var/lib/kubelet/pods/32025844-1d97-468f-b79d-638da9e02240/volumes" Oct 29 22:42:47.413548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:47.413516 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h"] Oct 29 22:42:47.413869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:47.413811 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" containerID="cri-o://58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158" gracePeriod=30 Oct 29 22:42:47.439008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:47.438977 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5"] Oct 29 22:42:47.439222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:47.439200 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" containerID="cri-o://b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea" gracePeriod=30 Oct 29 22:42:48.596644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:48.596605 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:42:48.597054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:48.596664 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:50.928256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:50.928218 5367 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:42:50.928747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:50.928297 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:52.332658 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:42:52.332623 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:42:52.802296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.801905 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:42:52.850107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.850078 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:42:52.907368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.907277 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.907682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.907656 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.907898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.907860 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.907924 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.907971 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908043 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908110 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908161 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908273 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908348 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908386 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908510 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908544 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908581 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908675 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908719 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.908952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908763 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data\") pod \"c7c362d6-ca73-4287-96e3-08b67ca51383\" (UID: \"c7c362d6-ca73-4287-96e3-08b67ca51383\") " Oct 29 22:42:52.908952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.908800 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq\") pod \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\" (UID: \"afb9022c-2c3f-43f4-a4e0-1ad32b282775\") " Oct 29 22:42:52.923770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.921152 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8"] Oct 29 22:42:52.942135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942096 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" Oct 29 22:42:52.942135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942132 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" Oct 29 22:42:52.942355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942150 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" Oct 29 22:42:52.942355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942157 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" Oct 29 22:42:52.942355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942224 5367 cpu_manager.go:414] "containerMap: removing container" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" Oct 29 22:42:52.942355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942233 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" Oct 29 22:42:52.942516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942452 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerName="istio-proxy" Oct 29 22:42:52.942516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942502 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" Oct 29 22:42:52.942516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.942511 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerName="istio-proxy" Oct 29 22:42:52.945427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.945391 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.946071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.946041 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.946271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.946250 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.946611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.946582 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.947383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.947165 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:52.948808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.947656 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:52.957739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.954630 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8"] Oct 29 22:42:52.957739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.956877 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:52.959841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.959529 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.959841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.959730 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.959841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.959743 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.959841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.959790 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "kube-api-access-qb7qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:52.959841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.959812 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.978320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.978263 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-63db1a-serving-cert\"" Oct 29 22:42:52.980064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.978667 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.984516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.981745 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9"] Oct 29 22:42:52.985943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.985915 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:52.995747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.989068 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:52.995747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.989068 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:52.995747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.989090 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:52.995747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.989236 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token" pod "c7c362d6-ca73-4287-96e3-08b67ca51383" (UID: "c7c362d6-ca73-4287-96e3-08b67ca51383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:52.995747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.989324 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "kube-api-access-d5vvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:52.995747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.994048 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo" pod "afb9022c-2c3f-43f4-a4e0-1ad32b282775" (UID: "afb9022c-2c3f-43f4-a4e0-1ad32b282775"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:52.996133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:52.996086 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9"] Oct 29 22:42:53.011425 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011394 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.011663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011641 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmb96\" (UniqueName: \"kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.011773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011758 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.011826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011812 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.011865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011850 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.011927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011869 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.011927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.011897 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.012057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012034 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.012139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012123 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.012244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012230 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.012300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012270 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.012345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012312 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gd6p\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.012345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012331 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.012526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012507 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012535 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c362d6-ca73-4287-96e3-08b67ca51383-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012549 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012563 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012575 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012588 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012600 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012611 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012627 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vvq\" (UniqueName: \"kubernetes.io/projected/afb9022c-2c3f-43f4-a4e0-1ad32b282775-kube-api-access-d5vvq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012640 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012653 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012665 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012678 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7qw\" (UniqueName: \"kubernetes.io/projected/c7c362d6-ca73-4287-96e3-08b67ca51383-kube-api-access-qb7qw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012690 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012724 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012737 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012749 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb9022c-2c3f-43f4-a4e0-1ad32b282775-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.012811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.012761 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c362d6-ca73-4287-96e3-08b67ca51383-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.113746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.113681 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmb96\" (UniqueName: \"kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.113931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.113782 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.113931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.113813 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.113931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.113862 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.113931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.113889 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.114167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.113948 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.114511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.114438 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.114923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.114832 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.114923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.114846 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.114923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.114873 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.115262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115234 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.115377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115316 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.115377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115361 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.115484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115393 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gd6p\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.115484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115423 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.115591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115529 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.115749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.115689 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.116195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.116122 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.116514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.116492 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.117159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.117125 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.117313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.117296 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.117639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.117618 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.118125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.118104 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.152717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.143338 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gd6p\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.152717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.145232 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmb96\" (UniqueName: \"kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96\") pod \"maas-api-e2e-api-b-63db1a-988f85d4-9x9r8\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.152717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.145719 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-p9rx9\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.236197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.236168 5367 generic.go:356] "Generic (PLEG): container finished" podID="c7c362d6-ca73-4287-96e3-08b67ca51383" containerID="58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158" exitCode=0 Oct 29 22:42:53.236351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.236251 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" Oct 29 22:42:53.236414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.236345 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" event={"ID":"c7c362d6-ca73-4287-96e3-08b67ca51383","Type":"ContainerDied","Data":"58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158"} Oct 29 22:42:53.236414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.236391 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h" event={"ID":"c7c362d6-ca73-4287-96e3-08b67ca51383","Type":"ContainerDied","Data":"e737af3de37cda77b2bb8e526d699c271993226cd073765dcdcd88afccb6853a"} Oct 29 22:42:53.236528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.236421 5367 scope.go:122] "RemoveContainer" containerID="58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158" Oct 29 22:42:53.238115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.238094 5367 generic.go:356] "Generic (PLEG): container finished" podID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" containerID="b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea" exitCode=0 Oct 29 22:42:53.238221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.238133 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" Oct 29 22:42:53.238221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.238169 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" event={"ID":"afb9022c-2c3f-43f4-a4e0-1ad32b282775","Type":"ContainerDied","Data":"b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea"} Oct 29 22:42:53.238345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.238219 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5" event={"ID":"afb9022c-2c3f-43f4-a4e0-1ad32b282775","Type":"ContainerDied","Data":"211f03a6c96e319509e934cafe260e9b61cbce0b8b2160ae786c5d29af8b7d5f"} Oct 29 22:42:53.249414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.249394 5367 scope.go:122] "RemoveContainer" containerID="79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88" Oct 29 22:42:53.290507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.290017 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:53.317721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.314382 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:53.374747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.368854 5367 scope.go:122] "RemoveContainer" containerID="58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158" Oct 29 22:42:53.394098 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:42:53.394061 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158\": container with ID starting with 58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158 not found: ID does not exist" containerID="58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158" Oct 29 22:42:53.394382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.394347 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158"} err="failed to get container status \"58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158\": rpc error: code = NotFound desc = could not find container \"58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158\": container with ID starting with 58a1cf7aaa5fa0eede42808eb82edced23f48359fd5247af9a83260b02394158 not found: ID does not exist" Oct 29 22:42:53.394616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.394599 5367 scope.go:122] "RemoveContainer" containerID="79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88" Oct 29 22:42:53.395250 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:42:53.395227 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88\": container with ID starting with 79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88 not found: ID does not exist" containerID="79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88" Oct 29 22:42:53.395406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.395381 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88"} err="failed to get container status \"79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88\": rpc error: code = NotFound desc = could not find container \"79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88\": container with ID starting with 79edae1402f7b59dd664d6f723748c74bde76a98448d00a70fa725dc670dce88 not found: ID does not exist" Oct 29 22:42:53.395555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.395540 5367 scope.go:122] "RemoveContainer" containerID="b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea" Oct 29 22:42:53.419939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.419872 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h"] Oct 29 22:42:53.433267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.433240 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-qs62h"] Oct 29 22:42:53.470923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.470826 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5"] Oct 29 22:42:53.490854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.490770 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-9m9w5"] Oct 29 22:42:53.528517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.528497 5367 scope.go:122] "RemoveContainer" containerID="b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea" Oct 29 22:42:53.531091 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:42:53.531002 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea\": container with ID starting with b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea not found: ID does not exist" containerID="b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea" Oct 29 22:42:53.531091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.531048 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea"} err="failed to get container status \"b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea\": rpc error: code = NotFound desc = could not find container \"b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea\": container with ID starting with b328e02d8b7d6372b9c8315d758a617cfe64cb2001acc258d63ffcfbbcfe3fea not found: ID does not exist" Oct 29 22:42:53.822931 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:42:53.822899 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e712137_2012_4419_90e9_95236293218a.slice/crio-b9ca9ea4a92d2dbf885a5b92f8d6ee5a0701a0853fd8a07cb1edcd581f9f749f WatchSource:0}: Error finding container b9ca9ea4a92d2dbf885a5b92f8d6ee5a0701a0853fd8a07cb1edcd581f9f749f: Status 404 returned error can't find the container with id b9ca9ea4a92d2dbf885a5b92f8d6ee5a0701a0853fd8a07cb1edcd581f9f749f Oct 29 22:42:53.823826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.823729 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8"] Oct 29 22:42:53.865883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.865847 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9"] Oct 29 22:42:53.872155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.872115 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:42:53.872248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.872206 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:42:53.872306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:53.872248 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:42:54.251791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.251754 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" event={"ID":"0e712137-2012-4419-90e9-95236293218a","Type":"ContainerStarted","Data":"9924244fee9753ea90eac4cde6c3be00f9d602dcdfcdc0ab05238205daf564e8"} Oct 29 22:42:54.251988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.251811 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" event={"ID":"0e712137-2012-4419-90e9-95236293218a","Type":"ContainerStarted","Data":"b9ca9ea4a92d2dbf885a5b92f8d6ee5a0701a0853fd8a07cb1edcd581f9f749f"} Oct 29 22:42:54.251988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.251864 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:42:54.254776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.254742 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" event={"ID":"31b187c6-0aa7-45eb-af5a-82f12ca47ae0","Type":"ContainerStarted","Data":"e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac"} Oct 29 22:42:54.254902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.254805 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" event={"ID":"31b187c6-0aa7-45eb-af5a-82f12ca47ae0","Type":"ContainerStarted","Data":"9ee9976c63006c6725a60c2993f9188cd85c3409e6350c302e06566e7c018a25"} Oct 29 22:42:54.295665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.295610 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" podStartSLOduration=32.979661293 podStartE2EDuration="33.295591926s" totalImagesPullingTime="315.930633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:54.293843852 +0000 UTC m=+1641.187918774" watchObservedRunningTime="2026-10-29 22:42:54.295591926 +0000 UTC m=+1641.189666845" Oct 29 22:42:54.315749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.315720 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:42:54.317049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.317026 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:42:54.317180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.317068 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 22:42:54.343196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:54.343140 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podStartSLOduration=63.343122962 podStartE2EDuration="1m3.343122962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:54.342876826 +0000 UTC m=+1641.236951748" watchObservedRunningTime="2026-10-29 22:42:54.343122962 +0000 UTC m=+1641.237197886" Oct 29 22:42:55.224565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:55.224536 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb9022c-2c3f-43f4-a4e0-1ad32b282775" path="/var/lib/kubelet/pods/afb9022c-2c3f-43f4-a4e0-1ad32b282775/volumes" Oct 29 22:42:55.225112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:55.225096 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c362d6-ca73-4287-96e3-08b67ca51383" path="/var/lib/kubelet/pods/c7c362d6-ca73-4287-96e3-08b67ca51383/volumes" Oct 29 22:42:55.316104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:55.316071 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:42:55.316277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:55.316122 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 22:42:56.315779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:56.315745 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:42:56.316183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:56.315796 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 22:42:57.316111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:57.316076 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:42:57.316500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:57.316128 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 22:42:58.315991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:58.315957 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:42:58.316176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:58.316013 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 22:42:58.411003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:58.410962 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56"] Oct 29 22:42:59.317795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:59.315438 5367 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:42:59.317795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:42:59.315525 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 22:43:00.301752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:00.294743 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:43:00.601892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:00.601863 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:43:00.602801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:00.602762 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:43:00.603484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:00.603465 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:43:03.414722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.412149 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" podUID="64910860-5fd8-4c44-8c90-15c9a17b4bd4" containerName="payload-processing" containerID="cri-o://6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca" gracePeriod=25 Oct 29 22:43:03.765664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.765634 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:43:03.877143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.876927 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z\" (UniqueName: \"kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z\") pod \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " Oct 29 22:43:03.877349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.877294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume\") pod \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\" (UID: \"64910860-5fd8-4c44-8c90-15c9a17b4bd4\") " Oct 29 22:43:03.878178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.878137 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume" pod "64910860-5fd8-4c44-8c90-15c9a17b4bd4" (UID: "64910860-5fd8-4c44-8c90-15c9a17b4bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:03.880155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.880126 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z" pod "64910860-5fd8-4c44-8c90-15c9a17b4bd4" (UID: "64910860-5fd8-4c44-8c90-15c9a17b4bd4"). InnerVolumeSpecName "kube-api-access-ljl5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:03.981002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.980919 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64910860-5fd8-4c44-8c90-15c9a17b4bd4-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:03.981002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:03.980956 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljl5z\" (UniqueName: \"kubernetes.io/projected/64910860-5fd8-4c44-8c90-15c9a17b4bd4-kube-api-access-ljl5z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:04.425157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.425121 5367 generic.go:356] "Generic (PLEG): container finished" podID="64910860-5fd8-4c44-8c90-15c9a17b4bd4" containerID="6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca" exitCode=0 Oct 29 22:43:04.425601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.425167 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" Oct 29 22:43:04.425601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.425178 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" event={"ID":"64910860-5fd8-4c44-8c90-15c9a17b4bd4","Type":"ContainerDied","Data":"6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca"} Oct 29 22:43:04.425601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.425231 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56" event={"ID":"64910860-5fd8-4c44-8c90-15c9a17b4bd4","Type":"ContainerDied","Data":"b650f30da31c65df1a465bb8787f5048fdd871a8028ada1b5b919afa555015b4"} Oct 29 22:43:04.425601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.425260 5367 scope.go:122] "RemoveContainer" containerID="6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca" Oct 29 22:43:04.441544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.441523 5367 scope.go:122] "RemoveContainer" containerID="6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca" Oct 29 22:43:04.441947 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:43:04.441923 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca\": container with ID starting with 6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca not found: ID does not exist" containerID="6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca" Oct 29 22:43:04.442028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.441958 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca"} err="failed to get container status \"6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca\": rpc error: code = NotFound desc = could not find container \"6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca\": container with ID starting with 6eddfb32831921c7d0f08b367d50d9bdb2ffd567704b390a55af7e1a1e7e3eca not found: ID does not exist" Oct 29 22:43:04.489121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.489086 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56"] Oct 29 22:43:04.501766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:04.501738 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-433712-7f84f4fdbd-sdj56"] Oct 29 22:43:05.233405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:05.233369 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64910860-5fd8-4c44-8c90-15c9a17b4bd4" path="/var/lib/kubelet/pods/64910860-5fd8-4c44-8c90-15c9a17b4bd4/volumes" Oct 29 22:43:05.532054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:05.531970 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj"] Oct 29 22:43:08.753861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:08.753816 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:10.532564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:10.532518 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" podUID="f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" containerName="payload-pre-processing" containerID="cri-o://2819bccd276f214497de9e8feedff3ec92f14de9a64acf09e07eba1d311237c3" gracePeriod=25 Oct 29 22:43:10.699648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:10.699617 5367 generic.go:356] "Generic (PLEG): container finished" podID="f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" containerID="2819bccd276f214497de9e8feedff3ec92f14de9a64acf09e07eba1d311237c3" exitCode=0 Oct 29 22:43:10.699862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:10.699720 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" event={"ID":"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8","Type":"ContainerDied","Data":"2819bccd276f214497de9e8feedff3ec92f14de9a64acf09e07eba1d311237c3"} Oct 29 22:43:10.902535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:10.902506 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:43:11.018453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.018412 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb\" (UniqueName: \"kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb\") pod \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " Oct 29 22:43:11.018679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.018531 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume\") pod \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\" (UID: \"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8\") " Oct 29 22:43:11.018952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.018922 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume" pod "f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" (UID: "f1b1a47b-add4-46a5-ae68-3c1d2f211ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:11.019405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.019385 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.021286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.021261 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb" pod "f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" (UID: "f1b1a47b-add4-46a5-ae68-3c1d2f211ab8"). InnerVolumeSpecName "kube-api-access-9q6nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:11.122343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.122256 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6nb\" (UniqueName: \"kubernetes.io/projected/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8-kube-api-access-9q6nb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.711173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.711140 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" event={"ID":"f1b1a47b-add4-46a5-ae68-3c1d2f211ab8","Type":"ContainerDied","Data":"f45a69656de4c65e8f55e22f04b93fafe6f4c05ee5f574a5698e3b67688f3a43"} Oct 29 22:43:11.711682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.711198 5367 scope.go:122] "RemoveContainer" containerID="2819bccd276f214497de9e8feedff3ec92f14de9a64acf09e07eba1d311237c3" Oct 29 22:43:11.711682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.711156 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj" Oct 29 22:43:11.784526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.784493 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj"] Oct 29 22:43:11.790317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:11.790289 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c06e05-6485ff7479-6mkhj"] Oct 29 22:43:13.228892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:13.226974 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" path="/var/lib/kubelet/pods/f1b1a47b-add4-46a5-ae68-3c1d2f211ab8/volumes" Oct 29 22:43:18.816852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.816818 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252"] Oct 29 22:43:18.833065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.817586 5367 cpu_manager.go:414] "containerMap: removing container" podUID="64910860-5fd8-4c44-8c90-15c9a17b4bd4" containerName="payload-processing" Oct 29 22:43:18.833065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.817600 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64910860-5fd8-4c44-8c90-15c9a17b4bd4" containerName="payload-processing" Oct 29 22:43:18.833065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.817641 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" containerName="payload-pre-processing" Oct 29 22:43:18.833065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.817646 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" containerName="payload-pre-processing" Oct 29 22:43:18.833065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.817771 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="64910860-5fd8-4c44-8c90-15c9a17b4bd4" containerName="payload-processing" Oct 29 22:43:18.833065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.817788 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b1a47b-add4-46a5-ae68-3c1d2f211ab8" containerName="payload-pre-processing" Oct 29 22:43:18.848240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.847018 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:18.853089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.853063 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252"] Oct 29 22:43:18.863403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.863355 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:43:18.876992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.876967 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:43:18.930851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.930820 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:18.931021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.930857 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrjb\" (UniqueName: \"kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:18.931086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:18.931034 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.032828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.032796 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.032995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.032904 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrjb\" (UniqueName: \"kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.033065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.033037 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.033743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.033718 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.035181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.035156 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.052797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.052770 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrjb\" (UniqueName: \"kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb\") pod \"maas-api-revoke-keys-e2e-api-a-1ae775-l4252\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.165437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.165363 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:19.466869 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:43:19.466837 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2262a60_83be_4f31_946e_aa4135a45e37.slice/crio-69ad96771459632d14b5446eff093b36cb65b63a8607685a53e7e1ac62e90c77 WatchSource:0}: Error finding container 69ad96771459632d14b5446eff093b36cb65b63a8607685a53e7e1ac62e90c77: Status 404 returned error can't find the container with id 69ad96771459632d14b5446eff093b36cb65b63a8607685a53e7e1ac62e90c77 Oct 29 22:43:19.467417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.467387 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252"] Oct 29 22:43:19.863316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.863283 5367 generic.go:356] "Generic (PLEG): container finished" podID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerID="51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21" exitCode=0 Oct 29 22:43:19.863825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.863354 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" event={"ID":"5ee79d16-2b7e-4129-8cff-8235bde26d3b","Type":"ContainerDied","Data":"51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21"} Oct 29 22:43:19.863915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.863893 5367 scope.go:122] "RemoveContainer" containerID="51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21" Oct 29 22:43:19.865517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.865488 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" event={"ID":"e2262a60-83be-4f31-946e-aa4135a45e37","Type":"ContainerStarted","Data":"4755850e2c465a597648af39d6dcca0e179d3ec934b0f39631756da6fc30c486"} Oct 29 22:43:19.865738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.865674 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" event={"ID":"e2262a60-83be-4f31-946e-aa4135a45e37","Type":"ContainerStarted","Data":"69ad96771459632d14b5446eff093b36cb65b63a8607685a53e7e1ac62e90c77"} Oct 29 22:43:19.866114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.866088 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:19.866199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.866161 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:19.866199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.866192 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:19.979011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:19.961196 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" podStartSLOduration=1.961178303 podStartE2EDuration="1.961178303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:19.960549969 +0000 UTC m=+1666.854624889" watchObservedRunningTime="2026-10-29 22:43:19.961178303 +0000 UTC m=+1666.855253223" Oct 29 22:43:20.888657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:20.888622 5367 generic.go:356] "Generic (PLEG): container finished" podID="e2262a60-83be-4f31-946e-aa4135a45e37" containerID="4755850e2c465a597648af39d6dcca0e179d3ec934b0f39631756da6fc30c486" exitCode=0 Oct 29 22:43:20.889070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:20.888711 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" event={"ID":"e2262a60-83be-4f31-946e-aa4135a45e37","Type":"ContainerDied","Data":"4755850e2c465a597648af39d6dcca0e179d3ec934b0f39631756da6fc30c486"} Oct 29 22:43:20.890957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:20.890934 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" event={"ID":"5ee79d16-2b7e-4129-8cff-8235bde26d3b","Type":"ContainerStarted","Data":"4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2"} Oct 29 22:43:21.688994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:21.688959 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:43:21.690459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:21.690423 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:43:21.690586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:21.690492 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:43:22.128650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.128624 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:22.211496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.211462 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca\") pod \"e2262a60-83be-4f31-946e-aa4135a45e37\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " Oct 29 22:43:22.211740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.211719 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb\" (UniqueName: \"kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb\") pod \"e2262a60-83be-4f31-946e-aa4135a45e37\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " Oct 29 22:43:22.212045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.212023 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca\") pod \"e2262a60-83be-4f31-946e-aa4135a45e37\" (UID: \"e2262a60-83be-4f31-946e-aa4135a45e37\") " Oct 29 22:43:22.212364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.212334 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca" pod "e2262a60-83be-4f31-946e-aa4135a45e37" (UID: "e2262a60-83be-4f31-946e-aa4135a45e37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:22.212976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.212948 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:22.213616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.213591 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca" pod "e2262a60-83be-4f31-946e-aa4135a45e37" (UID: "e2262a60-83be-4f31-946e-aa4135a45e37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:22.214208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.214187 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb" pod "e2262a60-83be-4f31-946e-aa4135a45e37" (UID: "e2262a60-83be-4f31-946e-aa4135a45e37"). InnerVolumeSpecName "kube-api-access-lkrjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:22.314949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.314919 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkrjb\" (UniqueName: \"kubernetes.io/projected/e2262a60-83be-4f31-946e-aa4135a45e37-kube-api-access-lkrjb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:22.314949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.314945 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2262a60-83be-4f31-946e-aa4135a45e37-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:22.332561 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:43:22.332527 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:43:22.689433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.689401 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:43:22.689642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.689452 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:43:22.883262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.883233 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:43:22.908773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.908727 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" event={"ID":"e2262a60-83be-4f31-946e-aa4135a45e37","Type":"ContainerDied","Data":"69ad96771459632d14b5446eff093b36cb65b63a8607685a53e7e1ac62e90c77"} Oct 29 22:43:22.908941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.908799 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ad96771459632d14b5446eff093b36cb65b63a8607685a53e7e1ac62e90c77" Oct 29 22:43:22.908941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:22.908741 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252" Oct 29 22:43:23.689406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:23.689366 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:43:23.689808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:23.689438 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:43:24.689267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:24.689233 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:43:24.689475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:24.689295 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:43:25.689414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:25.689377 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:43:25.689626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:25.689448 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:43:26.715940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:26.713426 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:43:26.730599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:26.730568 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:43:35.502107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:35.502074 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h"] Oct 29 22:43:35.502633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:35.502262 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" containerID="cri-o://bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61" gracePeriod=30 Oct 29 22:43:36.547438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:36.547409 5367 scope.go:122] "RemoveContainer" containerID="bb857e2a51acb41eb70d6228bb1c359768c3a60da1031352423da014536876b6" Oct 29 22:43:40.827674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.827647 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:43:40.891329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.891299 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k"] Oct 29 22:43:40.892090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.892074 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e2262a60-83be-4f31-946e-aa4135a45e37" containerName="revoke-keys" Oct 29 22:43:40.892141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.892091 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2262a60-83be-4f31-946e-aa4135a45e37" containerName="revoke-keys" Oct 29 22:43:40.892141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.892118 5367 cpu_manager.go:414] "containerMap: removing container" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" Oct 29 22:43:40.892141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.892123 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" Oct 29 22:43:40.892251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.892242 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2262a60-83be-4f31-946e-aa4135a45e37" containerName="revoke-keys" Oct 29 22:43:40.892290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.892256 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerName="istio-proxy" Oct 29 22:43:40.895986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.895970 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.916749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.915633 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k"] Oct 29 22:43:40.949673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.949638 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.949881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.949726 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.949881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.949824 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.949943 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.949943 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:40.950212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950190 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950242 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950283 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950342 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950378 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket\") pod \"26754557-c763-4176-b3dc-dbd3b34a9e7d\" (UID: \"26754557-c763-4176-b3dc-dbd3b34a9e7d\") " Oct 29 22:43:40.950863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950839 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.950970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.950891 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951055 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951112 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951188 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951259 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951257 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:40.951509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951485 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:40.951594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951542 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951682 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951763 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bn8b\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:40.951987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951967 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26754557-c763-4176-b3dc-dbd3b34a9e7d-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:40.952148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.951994 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:40.952148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.952016 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:40.952148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.952136 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:40.952432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.952321 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:40.952568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.952542 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:40.952878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.952854 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:40.954341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.954312 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "kube-api-access-dkptk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:40.954900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:40.954860 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token" pod "26754557-c763-4176-b3dc-dbd3b34a9e7d" (UID: "26754557-c763-4176-b3dc-dbd3b34a9e7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:41.053764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.053609 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.053764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.053678 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bn8b\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.053840 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.053878 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054008 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054051 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054118 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054184 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.054272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054237 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054373 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054394 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054418 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkptk\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-kube-api-access-dkptk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054434 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054446 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054463 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26754557-c763-4176-b3dc-dbd3b34a9e7d-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.054608 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.055046 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.056094 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.056223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.056112 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.056874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.056357 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.057647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.057620 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.059169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.058888 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.073731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.073693 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.073911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.073889 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bn8b\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.223479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.223239 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:41.320329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.320252 5367 generic.go:356] "Generic (PLEG): container finished" podID="26754557-c763-4176-b3dc-dbd3b34a9e7d" containerID="bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61" exitCode=0 Oct 29 22:43:41.320523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.320379 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" Oct 29 22:43:41.321411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.320398 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" event={"ID":"26754557-c763-4176-b3dc-dbd3b34a9e7d","Type":"ContainerDied","Data":"bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61"} Oct 29 22:43:41.321473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.321451 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h" event={"ID":"26754557-c763-4176-b3dc-dbd3b34a9e7d","Type":"ContainerDied","Data":"616b389bd1c91efe6a00ad5b6f7a47b72327b3c471642a91df3804eb0ab63c90"} Oct 29 22:43:41.321508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.321487 5367 scope.go:122] "RemoveContainer" containerID="bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61" Oct 29 22:43:41.337521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.337503 5367 scope.go:122] "RemoveContainer" containerID="bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61" Oct 29 22:43:41.337842 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:43:41.337814 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61\": container with ID starting with bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61 not found: ID does not exist" containerID="bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61" Oct 29 22:43:41.337958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.337845 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61"} err="failed to get container status \"bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61\": rpc error: code = NotFound desc = could not find container \"bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61\": container with ID starting with bb75e5ed0670802fdf3e72d6df3d69cc6ed9eb502a0e9c5581922c99f096db61 not found: ID does not exist" Oct 29 22:43:41.383596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.383569 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h"] Oct 29 22:43:41.394283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.394256 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-h5r7h"] Oct 29 22:43:41.554032 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:43:41.554005 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc578fbe_af4d_42d1_8da6_169c2e314e1e.slice/crio-401443bbf6bd1a0333d4b8ce075727fc0f3d96928ff5e32a384a9f56cf6e9017 WatchSource:0}: Error finding container 401443bbf6bd1a0333d4b8ce075727fc0f3d96928ff5e32a384a9f56cf6e9017: Status 404 returned error can't find the container with id 401443bbf6bd1a0333d4b8ce075727fc0f3d96928ff5e32a384a9f56cf6e9017 Oct 29 22:43:41.556121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.556094 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k"] Oct 29 22:43:41.557416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.557374 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:41.557509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.557465 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:41.557548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:41.557539 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:42.333880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.333842 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" event={"ID":"bc578fbe-af4d-42d1-8da6-169c2e314e1e","Type":"ContainerStarted","Data":"aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc"} Oct 29 22:43:42.334276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.333907 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" event={"ID":"bc578fbe-af4d-42d1-8da6-169c2e314e1e","Type":"ContainerStarted","Data":"401443bbf6bd1a0333d4b8ce075727fc0f3d96928ff5e32a384a9f56cf6e9017"} Oct 29 22:43:42.383961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.383911 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podStartSLOduration=18.383896817 podStartE2EDuration="18.383896817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:42.379373678 +0000 UTC m=+1689.273448599" watchObservedRunningTime="2026-10-29 22:43:42.383896817 +0000 UTC m=+1689.277971736" Oct 29 22:43:42.948485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.948440 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6"] Oct 29 22:43:42.953423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.953402 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:42.998135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.994432 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6"] Oct 29 22:43:42.998135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.994785 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:43:42.998135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:42.995049 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:43:43.095646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.095607 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ch2\" (UniqueName: \"kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.095889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.095694 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.095958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.095944 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.198010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.197974 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ch2\" (UniqueName: \"kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.198232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.198043 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.198232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.198173 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.198963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.198907 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.200666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.200640 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.218364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.218342 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ch2\" (UniqueName: \"kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.226184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.226155 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:43:43.226307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.226201 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 22:43:43.226458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.226438 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26754557-c763-4176-b3dc-dbd3b34a9e7d" path="/var/lib/kubelet/pods/26754557-c763-4176-b3dc-dbd3b34a9e7d/volumes" Oct 29 22:43:43.226809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.226795 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:43.300252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.300222 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:43.637741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:43.637675 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6"] Oct 29 22:43:43.638260 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:43:43.637835 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c14f492_1768_4c9e_86fe_770e1e207307.slice/crio-9fd56799ee5d62992f0f0cbee6aade2d44341fffbed320c2c6a86e81096c85a1 WatchSource:0}: Error finding container 9fd56799ee5d62992f0f0cbee6aade2d44341fffbed320c2c6a86e81096c85a1: Status 404 returned error can't find the container with id 9fd56799ee5d62992f0f0cbee6aade2d44341fffbed320c2c6a86e81096c85a1 Oct 29 22:43:44.225120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:44.225093 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:43:44.225243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:44.225139 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 22:43:44.399237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:44.399203 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" event={"ID":"0c14f492-1768-4c9e-86fe-770e1e207307","Type":"ContainerStarted","Data":"440c6e86f74f73cd42cd68f5d68511414d4897dc1b0e7f155a1feb28171b8f3c"} Oct 29 22:43:44.399385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:44.399254 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" event={"ID":"0c14f492-1768-4c9e-86fe-770e1e207307","Type":"ContainerStarted","Data":"9fd56799ee5d62992f0f0cbee6aade2d44341fffbed320c2c6a86e81096c85a1"} Oct 29 22:43:44.442322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:44.442216 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" podStartSLOduration=2.442203108 podStartE2EDuration="2.442203108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:44.439068166 +0000 UTC m=+1691.333143084" watchObservedRunningTime="2026-10-29 22:43:44.442203108 +0000 UTC m=+1691.336278065" Oct 29 22:43:45.224060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:45.223983 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:43:45.224060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:45.224032 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 22:43:45.410150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:45.410112 5367 generic.go:356] "Generic (PLEG): container finished" podID="0c14f492-1768-4c9e-86fe-770e1e207307" containerID="440c6e86f74f73cd42cd68f5d68511414d4897dc1b0e7f155a1feb28171b8f3c" exitCode=0 Oct 29 22:43:45.410365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:45.410191 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" event={"ID":"0c14f492-1768-4c9e-86fe-770e1e207307","Type":"ContainerDied","Data":"440c6e86f74f73cd42cd68f5d68511414d4897dc1b0e7f155a1feb28171b8f3c"} Oct 29 22:43:46.224329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.224297 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:43:46.224865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.224348 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 22:43:46.633896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.633862 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:46.798877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.797490 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca\") pod \"0c14f492-1768-4c9e-86fe-770e1e207307\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " Oct 29 22:43:46.798877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.797627 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca\") pod \"0c14f492-1768-4c9e-86fe-770e1e207307\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " Oct 29 22:43:46.798877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.797744 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2\" (UniqueName: \"kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2\") pod \"0c14f492-1768-4c9e-86fe-770e1e207307\" (UID: \"0c14f492-1768-4c9e-86fe-770e1e207307\") " Oct 29 22:43:46.799253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.799062 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca" pod "0c14f492-1768-4c9e-86fe-770e1e207307" (UID: "0c14f492-1768-4c9e-86fe-770e1e207307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:46.801464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.801436 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2" pod "0c14f492-1768-4c9e-86fe-770e1e207307" (UID: "0c14f492-1768-4c9e-86fe-770e1e207307"). InnerVolumeSpecName "kube-api-access-g7ch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:46.801753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.801664 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca" pod "0c14f492-1768-4c9e-86fe-770e1e207307" (UID: "0c14f492-1768-4c9e-86fe-770e1e207307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:46.900572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.900528 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:46.900572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.900566 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c14f492-1768-4c9e-86fe-770e1e207307-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:46.900863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:46.900582 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ch2\" (UniqueName: \"kubernetes.io/projected/0c14f492-1768-4c9e-86fe-770e1e207307-kube-api-access-g7ch2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:47.224171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:47.224139 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:43:47.224354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:47.224187 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 22:43:47.428756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:47.428725 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" Oct 29 22:43:47.428756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:47.428727 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6" event={"ID":"0c14f492-1768-4c9e-86fe-770e1e207307","Type":"ContainerDied","Data":"9fd56799ee5d62992f0f0cbee6aade2d44341fffbed320c2c6a86e81096c85a1"} Oct 29 22:43:47.428998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:47.428777 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd56799ee5d62992f0f0cbee6aade2d44341fffbed320c2c6a86e81096c85a1" Oct 29 22:43:48.224810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:48.224760 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:43:48.225407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:48.224848 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 22:43:49.344049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:49.344006 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:49.346308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:49.346286 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:49.352963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:49.352937 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:43:50.310821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:50.310787 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j"] Oct 29 22:43:50.311004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:50.310967 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" containerID="cri-o://bd06dcdc99e9023ee288ca2750b9dc6bc36d216f881ccc520c3f754322c9bc09" gracePeriod=30 Oct 29 22:43:52.332100 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:43:52.332060 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:43:53.455650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:53.455614 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc"] Oct 29 22:43:55.522541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.522509 5367 generic.go:356] "Generic (PLEG): container finished" podID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerID="bd06dcdc99e9023ee288ca2750b9dc6bc36d216f881ccc520c3f754322c9bc09" exitCode=0 Oct 29 22:43:55.523007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.522583 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" event={"ID":"1a6181a8-4db4-46bf-8c30-559987db96ae","Type":"ContainerDied","Data":"bd06dcdc99e9023ee288ca2750b9dc6bc36d216f881ccc520c3f754322c9bc09"} Oct 29 22:43:55.600692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.600626 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:43:55.686108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.686068 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn"] Oct 29 22:43:55.687373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.687326 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0c14f492-1768-4c9e-86fe-770e1e207307" containerName="revoke-keys" Oct 29 22:43:55.687373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.687353 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c14f492-1768-4c9e-86fe-770e1e207307" containerName="revoke-keys" Oct 29 22:43:55.687558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.687400 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" Oct 29 22:43:55.687558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.687408 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" Oct 29 22:43:55.687807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.687783 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" containerName="istio-proxy" Oct 29 22:43:55.687977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.687961 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c14f492-1768-4c9e-86fe-770e1e207307" containerName="revoke-keys" Oct 29 22:43:55.689093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689065 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689123 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689187 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689284 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689347 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689500 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689591 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689685 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.689805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.689755 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token\") pod \"1a6181a8-4db4-46bf-8c30-559987db96ae\" (UID: \"1a6181a8-4db4-46bf-8c30-559987db96ae\") " Oct 29 22:43:55.694216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.693475 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:55.694216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.693780 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.694216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.693966 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.695174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.695147 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:55.695411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.695389 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.695879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.695856 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.696124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.696102 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.698289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.698260 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.698380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.698306 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "kube-api-access-78jzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:55.699846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.699818 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo" pod "1a6181a8-4db4-46bf-8c30-559987db96ae" (UID: "1a6181a8-4db4-46bf-8c30-559987db96ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:55.709960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.709939 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn"] Oct 29 22:43:55.712225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.712204 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-139eb7-serving-cert\"" Oct 29 22:43:55.792127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792087 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.792363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792335 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.792566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792470 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.792662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792563 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5m4r\" (UniqueName: \"kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.792814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792784 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.792814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792802 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6181a8-4db4-46bf-8c30-559987db96ae-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.792814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792810 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.792814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792818 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.793155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792826 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.793155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792833 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.793155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792844 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78jzq\" (UniqueName: \"kubernetes.io/projected/1a6181a8-4db4-46bf-8c30-559987db96ae-kube-api-access-78jzq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.793155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792857 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6181a8-4db4-46bf-8c30-559987db96ae-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.793155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.792865 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6181a8-4db4-46bf-8c30-559987db96ae-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.893976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.893899 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.894122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.894053 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.894166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.894126 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.894205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.894175 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5m4r\" (UniqueName: \"kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.894869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.894842 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.896387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.896359 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.896478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.896397 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:55.914064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:55.914041 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5m4r\" (UniqueName: \"kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r\") pod \"maas-api-e2e-inf-b-139eb7-c46566579-j2gfn\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:56.052937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.052906 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:56.368039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.368010 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn"] Oct 29 22:43:56.368630 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:43:56.368606 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873f0eaf_50ea_4a57_8f20_cb6034928af5.slice/crio-564ee0e93ae8623fe94927636c129cb5046fd491d50ae7b86b3736f9b37f2e07 WatchSource:0}: Error finding container 564ee0e93ae8623fe94927636c129cb5046fd491d50ae7b86b3736f9b37f2e07: Status 404 returned error can't find the container with id 564ee0e93ae8623fe94927636c129cb5046fd491d50ae7b86b3736f9b37f2e07 Oct 29 22:43:56.538602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.538574 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" Oct 29 22:43:56.539121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.538574 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j" event={"ID":"1a6181a8-4db4-46bf-8c30-559987db96ae","Type":"ContainerDied","Data":"5923953edbf21aee02ca754f1afb1919b95850fd143e367425797c35c58fb447"} Oct 29 22:43:56.539121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.538765 5367 scope.go:122] "RemoveContainer" containerID="bd06dcdc99e9023ee288ca2750b9dc6bc36d216f881ccc520c3f754322c9bc09" Oct 29 22:43:56.540326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.540285 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" event={"ID":"873f0eaf-50ea-4a57-8f20-cb6034928af5","Type":"ContainerStarted","Data":"564ee0e93ae8623fe94927636c129cb5046fd491d50ae7b86b3736f9b37f2e07"} Oct 29 22:43:56.600098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.600070 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j"] Oct 29 22:43:56.613993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:56.613960 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-jvv6j"] Oct 29 22:43:57.292688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.291966 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6181a8-4db4-46bf-8c30-559987db96ae" path="/var/lib/kubelet/pods/1a6181a8-4db4-46bf-8c30-559987db96ae/volumes" Oct 29 22:43:57.548262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.548185 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x"] Oct 29 22:43:57.548665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.548373 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" podUID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" containerName="discovery" containerID="cri-o://aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef" gracePeriod=30 Oct 29 22:43:57.557181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.557143 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" event={"ID":"873f0eaf-50ea-4a57-8f20-cb6034928af5","Type":"ContainerStarted","Data":"3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b"} Oct 29 22:43:57.557330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.557269 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:43:57.636918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.636869 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" podStartSLOduration=9.291444561 podStartE2EDuration="9.636855046s" totalImagesPullingTime="345.410485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:57.635564873 +0000 UTC m=+1704.529639790" watchObservedRunningTime="2026-10-29 22:43:57.636855046 +0000 UTC m=+1704.530929965" Oct 29 22:43:57.907315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.907283 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:43:57.973855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.973619 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n"] Oct 29 22:43:57.975061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.974923 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" containerName="discovery" Oct 29 22:43:57.975061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.974948 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" containerName="discovery" Oct 29 22:43:57.975250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.975203 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" containerName="discovery" Oct 29 22:43:57.980274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.980251 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl"] Oct 29 22:43:57.980424 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.980391 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:57.986467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.985987 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd"] Oct 29 22:43:57.987757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.986925 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.004317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:57.999412 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr"] Oct 29 22:43:58.004317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.000795 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.026282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.026256 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q"] Oct 29 22:43:58.032757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.032731 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.048675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.048651 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.048859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.048688 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.049056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.049035 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.049134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.049118 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.049185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.049168 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:58.049449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.049423 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.049949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.049926 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.050042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.049998 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs\") pod \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\" (UID: \"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa\") " Oct 29 22:43:58.051257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.051232 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:58.053829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.053802 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.053933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.053855 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.053933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.053908 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.054068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.053937 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.054068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.053979 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.054068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054031 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.054220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054078 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5j2\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.054220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054120 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.054220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054177 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.054367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054220 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.054367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054286 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.054472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054390 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.054520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054474 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.054520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054503 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntxr\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.054626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054582 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.054679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054641 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.054761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054733 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.054820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054797 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.054875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054826 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.054950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054929 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.055008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054970 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sws7c\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.055069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.055069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055034 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.055069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055052 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.055226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055076 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.055226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055101 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.055226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055168 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.055378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055295 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.055378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.055306 5367 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-ca-configmap\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.055544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.054928 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "kube-api-access-rxgmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:58.056059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.056029 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:58.057517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.057497 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:58.060941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.060913 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:58.085484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.081409 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs" pod "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" (UID: "0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:58.085484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.084518 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n"] Oct 29 22:43:58.085484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.084565 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl"] Oct 29 22:43:58.085484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.084591 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd"] Oct 29 22:43:58.085484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.084685 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.091256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.091229 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr"] Oct 29 22:43:58.096808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.096779 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q"] Oct 29 22:43:58.156989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.156909 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.156989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.156946 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sws7c\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.156989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.156974 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.156994 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157058 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157121 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157165 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157185 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157209 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157236 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157310 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.157347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157341 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157387 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157423 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157444 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157448 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157550 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157650 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157681 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157756 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157804 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157820 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157850 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157860 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157879 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5j2\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157913 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.157928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157924 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.157971 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158009 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158055 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158098 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158142 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158197 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158249 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158272 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158284 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158295 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158097 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158538 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158611 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158617 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.158770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158726 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158779 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.158835 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntxr\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159071 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159099 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxmn\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159294 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159345 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159379 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159430 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqgn\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159463 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159493 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159555 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159580 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.159630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159606 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159671 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159744 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.159957 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160070 5367 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-cacerts\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160098 5367 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-local-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160119 5367 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-csr-dns-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160136 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgmv\" (UniqueName: \"kubernetes.io/projected/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-kube-api-access-rxgmv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160131 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160097 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160234 5367 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa-istio-kubeconfig\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.160344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160247 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.160688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160580 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.160749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160709 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.160921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.160899 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.161774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.161751 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.161862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.161799 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.177051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.177021 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.177593 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.177567 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sws7c\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.178018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.177996 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.178136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.178119 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.178179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.178121 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5j2\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.178889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.178872 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntxr\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.261327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261299 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261351 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261370 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261389 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261437 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261467 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261555 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261635 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261687 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261745 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261786 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261812 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261852 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.261877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261871 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261900 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxmn\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.261991 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262065 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqgn\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262098 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262159 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262162 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262211 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262674 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.262787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262745 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.262844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.262809 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.263048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.263033 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.263314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.263295 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.263468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.263453 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.264311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.264290 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.264430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.264361 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.264535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.264518 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.264938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.264923 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.282197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.282170 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.283154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.283135 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxmn\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.283235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.283172 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.283380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.283356 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqgn\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.331891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.331868 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:43:58.347193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.347163 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:43:58.352474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.352452 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:43:58.403076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.402976 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:43:58.416850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.415237 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:43:58.463172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.456423 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" podUID="8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" containerName="payload-pre-processing" containerID="cri-o://b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e" gracePeriod=25 Oct 29 22:43:58.570396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.570372 5367 generic.go:356] "Generic (PLEG): container finished" podID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" containerID="aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef" exitCode=0 Oct 29 22:43:58.570812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.570770 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" event={"ID":"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa","Type":"ContainerDied","Data":"aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef"} Oct 29 22:43:58.570885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.570815 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" event={"ID":"0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa","Type":"ContainerDied","Data":"3f0777ce13073bad733e886e6e64f855fe2dc43bb7445658f04808f48a6d013a"} Oct 29 22:43:58.570885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.570841 5367 scope.go:122] "RemoveContainer" containerID="aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef" Oct 29 22:43:58.571132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.571105 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x" Oct 29 22:43:58.632878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.630049 5367 scope.go:122] "RemoveContainer" containerID="aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef" Oct 29 22:43:58.632878 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:43:58.630483 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef\": container with ID starting with aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef not found: ID does not exist" containerID="aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef" Oct 29 22:43:58.632878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.630518 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef"} err="failed to get container status \"aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef\": rpc error: code = NotFound desc = could not find container \"aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef\": container with ID starting with aa3e4bd8fa9cdb4985ff4a209012698259da8b210e5f468979da5cd044be51ef not found: ID does not exist" Oct 29 22:43:58.698088 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:43:58.688022 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f992356_17a9_4281_943f_a567571fd9ef.slice/crio-60b7547df58e5d8bad983e34e6b70bb894d06e6918f5dd19aca86fed7bf88c40 WatchSource:0}: Error finding container 60b7547df58e5d8bad983e34e6b70bb894d06e6918f5dd19aca86fed7bf88c40: Status 404 returned error can't find the container with id 60b7547df58e5d8bad983e34e6b70bb894d06e6918f5dd19aca86fed7bf88c40 Oct 29 22:43:58.698088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.697221 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x"] Oct 29 22:43:58.698318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.698166 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.698318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.698251 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.698318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.698294 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.706005 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:43:58.705959 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97d9d81_60f1_47f2_98fb_997c8c4de79d.slice/crio-9b5eebd1eeb812e9a5b518336baa7b2a1185d585bf7087d0db23b4c6ba038ee2 WatchSource:0}: Error finding container 9b5eebd1eeb812e9a5b518336baa7b2a1185d585bf7087d0db23b4c6ba038ee2: Status 404 returned error can't find the container with id 9b5eebd1eeb812e9a5b518336baa7b2a1185d585bf7087d0db23b4c6ba038ee2 Oct 29 22:43:58.715053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.714898 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl"] Oct 29 22:43:58.761305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.761262 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.761632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.761603 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.761894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.761856 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.768787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.768196 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n"] Oct 29 22:43:58.768787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.768307 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6q2x"] Oct 29 22:43:58.768787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.768397 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd"] Oct 29 22:43:58.772854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.772762 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.772996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.772858 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.772996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.772902 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:58.930973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.930942 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:43:58.986295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.986258 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume\") pod \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " Oct 29 22:43:59.009614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.008536 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:59.009614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.008633 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:59.009614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.008675 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:59.010289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:58.986781 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume" pod "8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" (UID: "8e4520c6-0cc3-4c4f-b387-0d63d9f757fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:59.018361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.017049 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592\" (UniqueName: \"kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592\") pod \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\" (UID: \"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc\") " Oct 29 22:43:59.018361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.018107 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:59.068504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.045000 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr"] Oct 29 22:43:59.068504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.046994 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592" pod "8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" (UID: "8e4520c6-0cc3-4c4f-b387-0d63d9f757fc"). InnerVolumeSpecName "kube-api-access-rd592". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:59.068504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.059687 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q"] Oct 29 22:43:59.068504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.064671 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:59.068504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.064766 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:59.068504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.064840 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:43:59.120842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.120814 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd592\" (UniqueName: \"kubernetes.io/projected/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc-kube-api-access-rd592\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:43:59.318298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.306112 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa" path="/var/lib/kubelet/pods/0a9f0e48-c6a2-4e71-bcd0-f203ff7ff1aa/volumes" Oct 29 22:43:59.666731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.665425 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" event={"ID":"f51b751f-722b-4001-95cd-04ce87fd5356","Type":"ContainerStarted","Data":"2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0"} Oct 29 22:43:59.666731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.665497 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" event={"ID":"f51b751f-722b-4001-95cd-04ce87fd5356","Type":"ContainerStarted","Data":"d37d2f32f424d51c5ca52c9cec87dbf611f6e425539fb8efaa230b881490d304"} Oct 29 22:43:59.728486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.728422 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" event={"ID":"d1e03932-f445-49d2-85dc-6f89dd2657dc","Type":"ContainerStarted","Data":"ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97"} Oct 29 22:43:59.742873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.742680 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" event={"ID":"d1e03932-f445-49d2-85dc-6f89dd2657dc","Type":"ContainerStarted","Data":"fc9410ca632d458ef930e012d6ce91c0241245d44b936a5834f336aaaf3b61c4"} Oct 29 22:43:59.746763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.743438 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" event={"ID":"5f992356-17a9-4281-943f-a567571fd9ef","Type":"ContainerStarted","Data":"07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf"} Oct 29 22:43:59.746763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.743493 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" event={"ID":"5f992356-17a9-4281-943f-a567571fd9ef","Type":"ContainerStarted","Data":"60b7547df58e5d8bad983e34e6b70bb894d06e6918f5dd19aca86fed7bf88c40"} Oct 29 22:43:59.746763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.729369 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podStartSLOduration=35.7293505 podStartE2EDuration="35.7293505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:59.719988104 +0000 UTC m=+1706.614063024" watchObservedRunningTime="2026-10-29 22:43:59.7293505 +0000 UTC m=+1706.623425420" Oct 29 22:43:59.753232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.753202 5367 generic.go:356] "Generic (PLEG): container finished" podID="8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" containerID="b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e" exitCode=0 Oct 29 22:43:59.753499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.753479 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" Oct 29 22:43:59.753631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.753378 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" event={"ID":"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc","Type":"ContainerDied","Data":"b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e"} Oct 29 22:43:59.753803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.753785 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc" event={"ID":"8e4520c6-0cc3-4c4f-b387-0d63d9f757fc","Type":"ContainerDied","Data":"2ff9d0e1ddbc5a1a24be3b58d746253d5208af1982fc6bc2b03c33586fc4a5e0"} Oct 29 22:43:59.753959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.753943 5367 scope.go:122] "RemoveContainer" containerID="b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e" Oct 29 22:43:59.829881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.829815 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podStartSLOduration=41.829787467 podStartE2EDuration="41.829787467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:59.815248678 +0000 UTC m=+1706.709323602" watchObservedRunningTime="2026-10-29 22:43:59.829787467 +0000 UTC m=+1706.723862387" Oct 29 22:43:59.831398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.831378 5367 scope.go:122] "RemoveContainer" containerID="b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e" Oct 29 22:43:59.853672 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:43:59.853636 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e\": container with ID starting with b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e not found: ID does not exist" containerID="b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e" Oct 29 22:43:59.853978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.853943 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e"} err="failed to get container status \"b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e\": rpc error: code = NotFound desc = could not find container \"b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e\": container with ID starting with b6c387f2f1888850113e09d733217b2512f8ffa932310a252c043b5e0e47272e not found: ID does not exist" Oct 29 22:43:59.854501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.854471 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" event={"ID":"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424","Type":"ContainerStarted","Data":"a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056"} Oct 29 22:43:59.854675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.854661 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" event={"ID":"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424","Type":"ContainerStarted","Data":"23a80716d4dc0567958654f4e873f66c98a46dc71f1d56ae13dcc2ec800dcbc3"} Oct 29 22:43:59.881779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.875371 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" event={"ID":"d97d9d81-60f1-47f2-98fb-997c8c4de79d","Type":"ContainerStarted","Data":"6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c"} Oct 29 22:43:59.913854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.913802 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" event={"ID":"d97d9d81-60f1-47f2-98fb-997c8c4de79d","Type":"ContainerStarted","Data":"9b5eebd1eeb812e9a5b518336baa7b2a1185d585bf7087d0db23b4c6ba038ee2"} Oct 29 22:43:59.922295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:43:59.920075 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podStartSLOduration=35.920057272 podStartE2EDuration="35.920057272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:59.873283781 +0000 UTC m=+1706.767358701" watchObservedRunningTime="2026-10-29 22:43:59.920057272 +0000 UTC m=+1706.814132192" Oct 29 22:44:00.027871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.022944 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc"] Oct 29 22:44:00.037693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.036093 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d478b36e-6d476fb4f5-276bc"] Oct 29 22:44:00.052993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.052438 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podStartSLOduration=42.052421414 podStartE2EDuration="42.052421414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:00.037574079 +0000 UTC m=+1706.931648999" watchObservedRunningTime="2026-10-29 22:44:00.052421414 +0000 UTC m=+1706.946496335" Oct 29 22:44:00.350725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.341257 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:44:00.350725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.342833 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.350725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.342883 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:00.354725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.353217 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:44:00.358720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.357771 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:44:00.375725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.360772 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.375725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.360833 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:00.375725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.366831 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.375725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.366885 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:00.411722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.411412 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:44:00.411722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.411534 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.411722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.411576 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:00.417248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.417030 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:44:00.417248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.417145 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.417248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:00.417188 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:01.337851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.337794 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:44:01.431722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.402909 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.410188 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.457188 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.410446 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.457251 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.410965 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.457295 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.426173 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.457340 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.467894 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.482719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.467972 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:01.548725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.514806 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:01.548725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.526158 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" path="/var/lib/kubelet/pods/8e4520c6-0cc3-4c4f-b387-0d63d9f757fc/volumes" Oct 29 22:44:01.548725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.526962 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:01.657106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.656691 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:44:01.696922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.696847 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:01.746834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:01.746785 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.102894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.093353 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:44:02.113085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.098601 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:44:02.124722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.106480 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:44:02.154723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.106536 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.173848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.173806 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:02.447917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.447819 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:44:02.632161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.632128 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:44:02.633557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.449834 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podStartSLOduration=38.449812442 podStartE2EDuration="38.449812442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:00.142263001 +0000 UTC m=+1707.036337922" watchObservedRunningTime="2026-10-29 22:44:02.449812442 +0000 UTC m=+1709.343887380" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.699879 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.515796 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.451457 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.582264 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.451386 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.702902 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.451426 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.702970 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.452205 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.703025 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.514984 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.703060 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.536486 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.704894 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw"] Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.711205 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.732782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.711247 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:02.757725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.739016 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" containerName="payload-pre-processing" Oct 29 22:44:02.757725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.739056 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" containerName="payload-pre-processing" Oct 29 22:44:02.757725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.739333 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4520c6-0cc3-4c4f-b387-0d63d9f757fc" containerName="payload-pre-processing" Oct 29 22:44:02.757725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.739325 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:44:02.757725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.748314 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:02.770166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.770133 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:44:02.770430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.770403 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:44:02.770587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.770574 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:44:02.770724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.770711 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:44:02.770869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.770859 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:44:02.771784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.771507 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:02.791000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.790722 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-379a6d\"" Oct 29 22:44:02.928720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.903237 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:02.930719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:02.929613 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldk5\" (UniqueName: \"kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5\") pod \"payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:03.096723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.090623 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw"] Oct 29 22:44:03.115723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.112265 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:03.119856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.116241 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldk5\" (UniqueName: \"kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5\") pod \"payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:03.134926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.123780 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:44:03.140948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.140919 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:44:03.177851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.177802 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:03.178772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.178736 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:03.186967 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.186931 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:03.237727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.196182 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:44:03.237727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.196224 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl"] Oct 29 22:44:03.237727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.214733 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:03.372780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.346580 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:03.413822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.383778 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Oct 29 22:44:03.413822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.387869 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:03.413822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.399275 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:03.511722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.486809 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-379a6d-dockercfg-zvrlc\"" Oct 29 22:44:03.861782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.801508 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:04.529862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.529825 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:04.530351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.881058 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:44:04.688734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.872818 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 29 22:44:04.689153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.949683 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:04.689592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.949837 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:44:04.712008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.949926 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:44:04.729806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.033828 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:04.730326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.060286 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldk5\" (UniqueName: \"kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5\") pod \"payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:04.731863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.220926 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:04.743723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.221394 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.221515 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.345789 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.414980 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:04.791545 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.481s" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.791722 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.791799 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.791856 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.791879 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl"] Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.791973 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.792049 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:44:04.894820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.820867 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:04.919328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.915256 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:04.919328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.915679 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qk2\" (UniqueName: \"kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2\") pod \"payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:04.949882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.906250 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:05.024354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:03.957172 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.024750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.024716 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:05.062005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.048361 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:44:05.062005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.056643 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.062005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.056733 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.081986 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.082065 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.062491 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.113292 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.102998 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.119739 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:04.696566 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.120755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.119940 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:05.414747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.398187 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 29 22:44:05.520756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.502162 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qk2\" (UniqueName: \"kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2\") pod \"payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:05.520756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.502372 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:05.520756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.503441 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:05.563939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.558824 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:05.601004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.570443 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:44:05.673064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.638726 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.673064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.638781 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:05.673064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.642173 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.673064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.642217 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:05.679084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.679035 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.679327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.679299 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:05.679798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.679771 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.734369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.729531 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:05.734369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.696951 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:05.819265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.819224 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": context deadline exceeded" Oct 29 22:44:05.850597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.850565 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qk2\" (UniqueName: \"kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2\") pod \"payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:05.939079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.938965 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:06.007862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.007811 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.102563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:05.939339 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.132724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049757 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:44:06.329957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049818 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:44:06.329957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049863 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:44:06.329957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049883 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:06.329957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049904 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:44:06.329957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049922 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:44:06.329957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.049941 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:44:06.440116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.438976 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:44:06.440116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.439038 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.481777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.481741 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:06.482090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.482044 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:06.489469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.489436 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:44:06.489651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.489616 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.575930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.575871 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.651110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.613413 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:06.651564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.651342 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:06.846750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.670385 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.889739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.886805 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": context deadline exceeded" Oct 29 22:44:06.945611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.648965 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:44:06.945611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.894978 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:06.945611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.812220 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:06.945611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.896051 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:06.945611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.923043 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:06.945611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.923110 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:06.956539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.956507 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:06.957134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.956743 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:06.969755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:06.962397 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:07.168728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.159010 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:07.194723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.187801 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:44:07.201653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.201569 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:44:07.202557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.202524 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:44:07.203387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.203363 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:44:07.213814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.213784 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:07.214284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.214258 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:07.215173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.215141 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:44:07.330758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.319550 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:07.369067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.369031 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 22:44:07.369320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.369291 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 22:44:07.377724 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:07.370600 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.143s" Oct 29 22:44:07.378673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.378653 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:44:07.421065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.401950 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:07.422530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.422496 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:07.422742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.413235 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:07.422861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.422844 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:07.422973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.419797 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 22:44:07.423064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.423050 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 22:44:07.463938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.463846 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:07.470008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.469873 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:07.521541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.521491 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:07.691954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:07.667441 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 29 22:44:08.027785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.023062 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Oct 29 22:44:08.080502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.078986 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:44:08.200805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.200748 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.202691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.202657 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.313818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.306880 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 29 22:44:08.313818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.209844 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.313818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.209803 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:44:08.320816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.250409 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.364561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.250486 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:44:08.364850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.250504 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:08.365000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.250467 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.365144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.330821 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 29 22:44:08.412617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.412480 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krfjp" podUID="fd4be442-0fd5-483a-a2fb-45561dfdaafc" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.651549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.651463 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:08.651817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.651780 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:08.679722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.675874 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.160:50052: i/o timeout" Oct 29 22:44:08.761722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.734834 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:08.903071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.805221 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 29 22:44:08.903071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.805279 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 29 22:44:08.954732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:08.954179 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.112831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.112793 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:09.113090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.113061 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:09.275723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.259780 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 29 22:44:09.309858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.309103 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.315725 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:09.315471 5367 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.896s" Oct 29 22:44:09.336133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.336094 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.368571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.366011 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.368571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.367000 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.368571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.367289 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:44:09.368571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.367320 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.368571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.367455 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.378591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.378553 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.400902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.391102 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.436468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.436434 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:44:09.436718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.436677 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 29 22:44:09.472613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.472561 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:09.495632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.495580 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:44:09.495859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.495679 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.543676 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" event={"ID":"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a","Type":"ContainerStarted","Data":"f1f3e920c7c4b17d9cc8e31159f4b1a76edd41aaa244cd9ca7ae47c98386fde4"} Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.543873 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.543916 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544024 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544102 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544158 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544202 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544230 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544279 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544308 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544336 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544353 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br"] Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544385 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6"] Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544428 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544458 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544485 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544513 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:44:09.545038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.544541 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:44:09.547561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.547538 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:44:09.547685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.547584 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:44:09.547685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.547643 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" containerID="cri-o://6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c" gracePeriod=30 Oct 29 22:44:09.549908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.549857 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" containerID="cri-o://c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452" gracePeriod=30 Oct 29 22:44:09.552092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.552070 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:44:09.552196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.552155 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:44:09.552261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.552202 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:44:09.554066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.553403 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:44:09.641913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.641880 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw"] Oct 29 22:44:09.648060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:09.648034 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl"] Oct 29 22:44:10.338111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.338082 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:44:10.338519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.338355 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:44:10.339133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.339113 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:44:10.408753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.408679 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:44:10.409976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.409140 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:44:10.410806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.410786 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:44:10.548580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.548542 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" event={"ID":"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a","Type":"ContainerStarted","Data":"3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e"} Oct 29 22:44:10.548790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.548652 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:10.550944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.550920 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" event={"ID":"a4affabc-bf17-4f8e-a803-5a73d5485724","Type":"ContainerStarted","Data":"5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496"} Oct 29 22:44:10.551045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.550958 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" event={"ID":"a4affabc-bf17-4f8e-a803-5a73d5485724","Type":"ContainerStarted","Data":"c75794eb05c27ae159ae66f42332d3727b768cd092a9d7b1a6976c49f8b71220"} Oct 29 22:44:10.551045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.551007 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:10.592679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.592596 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" podStartSLOduration=8.592583644 podStartE2EDuration="8.592583644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:10.588514748 +0000 UTC m=+1717.482589665" watchObservedRunningTime="2026-10-29 22:44:10.592583644 +0000 UTC m=+1717.486658561" Oct 29 22:44:10.635149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:10.635096 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" podStartSLOduration=8.635079934 podStartE2EDuration="8.635079934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:10.627780731 +0000 UTC m=+1717.521855652" watchObservedRunningTime="2026-10-29 22:44:10.635079934 +0000 UTC m=+1717.529154854" Oct 29 22:44:11.907722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:11.901307 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6"] Oct 29 22:44:11.910762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:11.910193 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-rqnf6"] Oct 29 22:44:13.226904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:13.226870 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c14f492-1768-4c9e-86fe-770e1e207307" path="/var/lib/kubelet/pods/0c14f492-1768-4c9e-86fe-770e1e207307/volumes" Oct 29 22:44:14.862267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.862236 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:44:14.886914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.886890 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:44:14.911834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.911798 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.911955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.911898 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.911955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.911926 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912007 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912017 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:14.912277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912253 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912267 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:14.912414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912288 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912327 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912381 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912543 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:14.912567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912548 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs\") pod \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\" (UID: \"17e7493c-6aaa-4d36-b404-d806b0c7d1cf\") " Oct 29 22:44:14.912833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912795 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:14.912950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.912858 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:14.913629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.913601 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:14.913755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.913634 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:14.913755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.913650 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:14.913755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.913667 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:14.913755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.913681 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:14.914300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.914280 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:14.914393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.914298 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:14.914393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.914378 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:14.914492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:14.914478 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678" pod "17e7493c-6aaa-4d36-b404-d806b0c7d1cf" (UID: "17e7493c-6aaa-4d36-b404-d806b0c7d1cf"). InnerVolumeSpecName "kube-api-access-gj678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:15.014267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014229 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014348 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014402 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014458 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014543 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014629 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014734 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.014813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014693 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:15.014946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014927 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.015005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.014991 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo\") pod \"35f9fd72-e4e7-425b-bd9d-2057366f47da\" (UID: \"35f9fd72-e4e7-425b-bd9d-2057366f47da\") " Oct 29 22:44:15.015059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015030 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:15.015059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015044 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:15.015258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015229 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:15.015938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015911 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f9fd72-e4e7-425b-bd9d-2057366f47da-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015943 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015957 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015969 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015982 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.015995 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.016008 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.016021 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj678\" (UniqueName: \"kubernetes.io/projected/17e7493c-6aaa-4d36-b404-d806b0c7d1cf-kube-api-access-gj678\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.016785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.016758 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:15.018102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.018078 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "kube-api-access-xwmkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:15.018321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.018294 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:15.018432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.018325 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:15.018902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.018883 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token" pod "35f9fd72-e4e7-425b-bd9d-2057366f47da" (UID: "35f9fd72-e4e7-425b-bd9d-2057366f47da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:15.117913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.117838 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmkd\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-kube-api-access-xwmkd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.117913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.117864 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.117913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.117873 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.117913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.117882 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.117913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.117891 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f9fd72-e4e7-425b-bd9d-2057366f47da-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:15.604662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.604632 5367 generic.go:356] "Generic (PLEG): container finished" podID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerID="6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c" exitCode=0 Oct 29 22:44:15.604853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.604679 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" Oct 29 22:44:15.604853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.604735 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" event={"ID":"35f9fd72-e4e7-425b-bd9d-2057366f47da","Type":"ContainerDied","Data":"6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c"} Oct 29 22:44:15.604853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.604795 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6" event={"ID":"35f9fd72-e4e7-425b-bd9d-2057366f47da","Type":"ContainerDied","Data":"fdd96f65d3805dc406926586f89f33c337e553ce56e862ac27f032ac9a3d63bd"} Oct 29 22:44:15.604853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.604817 5367 scope.go:122] "RemoveContainer" containerID="6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c" Oct 29 22:44:15.607976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.607956 5367 generic.go:356] "Generic (PLEG): container finished" podID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerID="c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452" exitCode=0 Oct 29 22:44:15.608065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.608015 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" Oct 29 22:44:15.608103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.608016 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" event={"ID":"17e7493c-6aaa-4d36-b404-d806b0c7d1cf","Type":"ContainerDied","Data":"c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452"} Oct 29 22:44:15.608141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.608132 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br" event={"ID":"17e7493c-6aaa-4d36-b404-d806b0c7d1cf","Type":"ContainerDied","Data":"1184dab3b7fcd6c1227c1be6149a1c7e3e803ecfbfffba192833ed8a8602b7d7"} Oct 29 22:44:15.617110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.617092 5367 scope.go:122] "RemoveContainer" containerID="6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c" Oct 29 22:44:15.617387 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:15.617367 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c\": container with ID starting with 6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c not found: ID does not exist" containerID="6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c" Oct 29 22:44:15.617432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.617394 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c"} err="failed to get container status \"6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c\": rpc error: code = NotFound desc = could not find container \"6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c\": container with ID starting with 6654889386dcc303a07c6e6c0e9658df0b34e38c0de0c7fc1a7f9f1e39c7d18c not found: ID does not exist" Oct 29 22:44:15.617432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.617412 5367 scope.go:122] "RemoveContainer" containerID="c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452" Oct 29 22:44:15.629056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.628418 5367 scope.go:122] "RemoveContainer" containerID="c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452" Oct 29 22:44:15.630040 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:15.629941 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452\": container with ID starting with c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452 not found: ID does not exist" containerID="c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452" Oct 29 22:44:15.630040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.629971 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452"} err="failed to get container status \"c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452\": rpc error: code = NotFound desc = could not find container \"c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452\": container with ID starting with c99aa47a20a55628bd2eeb400c6a9aa5ceefaf353e30b9ed854791c52ec84452 not found: ID does not exist" Oct 29 22:44:15.662189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.662166 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6"] Oct 29 22:44:15.675474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.675448 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-p9hg6"] Oct 29 22:44:15.705487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.705459 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br"] Oct 29 22:44:15.710388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:15.710367 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-6k2br"] Oct 29 22:44:17.225527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:17.225498 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" path="/var/lib/kubelet/pods/17e7493c-6aaa-4d36-b404-d806b0c7d1cf/volumes" Oct 29 22:44:17.226187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:17.226172 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" path="/var/lib/kubelet/pods/35f9fd72-e4e7-425b-bd9d-2057366f47da/volumes" Oct 29 22:44:18.977918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:18.977874 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9"] Oct 29 22:44:18.978805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:18.978771 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" containerID="cri-o://e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac" gracePeriod=30 Oct 29 22:44:18.998172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:18.998138 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4"] Oct 29 22:44:18.998660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:18.998633 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" containerID="cri-o://ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc" gracePeriod=30 Oct 29 22:44:20.183303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.183269 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp"] Oct 29 22:44:20.184015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.183981 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" containerID="cri-o://9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970" gracePeriod=30 Oct 29 22:44:20.351137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.351100 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld"] Oct 29 22:44:20.352408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.352378 5367 cpu_manager.go:414] "containerMap: removing container" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" Oct 29 22:44:20.352408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.352407 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" Oct 29 22:44:20.352610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.352452 5367 cpu_manager.go:414] "containerMap: removing container" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" Oct 29 22:44:20.352610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.352460 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" Oct 29 22:44:20.352776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.352756 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f9fd72-e4e7-425b-bd9d-2057366f47da" containerName="istio-proxy" Oct 29 22:44:20.352837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.352788 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e7493c-6aaa-4d36-b404-d806b0c7d1cf" containerName="istio-proxy" Oct 29 22:44:20.357441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.357422 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.377169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.377143 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld"] Oct 29 22:44:20.382746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.380889 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6bc7b2-openshift-default-dockercfg-g8djs\"" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.778770 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.778809 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.778828 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.778920 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.778998 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.779024 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zpv\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.779068 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.779090 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.780217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.779121 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.870873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.870844 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt"] Oct 29 22:44:20.871324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.871291 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" containerID="cri-o://c6482d182bf5ce9a3fa93defa69b0e36d23a33a9ae8270e5f53a06950b797f30" gracePeriod=30 Oct 29 22:44:20.881097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881072 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881177 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881208 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zpv\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881265 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881297 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881327 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881529 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881554 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881607 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.881722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.881637 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.889002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.882758 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.889002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.883002 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.889002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.886321 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.889002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.886693 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.891388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.891366 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.892969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.892936 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.901510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.901489 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.904220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.904194 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zpv\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:20.974973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:20.974943 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:21.095322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.095286 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk"] Oct 29 22:44:21.095579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.095530 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" containerID="cri-o://4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841" gracePeriod=30 Oct 29 22:44:21.306474 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:21.306447 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode858318b_4888_4949_8200_ccc0e1719ec6.slice/crio-d97b9dd6757f9ea783a4f9e6aad75ce7bfe0cd11d5fa564e08a7236136f86318 WatchSource:0}: Error finding container d97b9dd6757f9ea783a4f9e6aad75ce7bfe0cd11d5fa564e08a7236136f86318: Status 404 returned error can't find the container with id d97b9dd6757f9ea783a4f9e6aad75ce7bfe0cd11d5fa564e08a7236136f86318 Oct 29 22:44:21.307991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.307968 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld"] Oct 29 22:44:21.309791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.309693 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:44:21.309892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.309842 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:44:21.309960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.309891 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:44:21.597719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.594407 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:44:21.597719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.597351 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:44:21.879794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.879687 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" event={"ID":"e858318b-4888-4949-8200-ccc0e1719ec6","Type":"ContainerStarted","Data":"cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938"} Oct 29 22:44:21.879794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.879778 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" event={"ID":"e858318b-4888-4949-8200-ccc0e1719ec6","Type":"ContainerStarted","Data":"d97b9dd6757f9ea783a4f9e6aad75ce7bfe0cd11d5fa564e08a7236136f86318"} Oct 29 22:44:21.952754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.952564 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podStartSLOduration=1.952546407 podStartE2EDuration="1.952546407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:21.935822258 +0000 UTC m=+1728.829897191" watchObservedRunningTime="2026-10-29 22:44:21.952546407 +0000 UTC m=+1728.846621327" Oct 29 22:44:21.975965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:21.975769 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:22.334954 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:22.334916 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:44:22.651859 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.651777 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x"] Oct 29 22:44:22.657201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.657167 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.685031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.685003 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x"] Oct 29 22:44:22.691080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.691045 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-172c10-openshift-default-dockercfg-sbfcd\"" Oct 29 22:44:22.725041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725015 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725124 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725188 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725265 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725282 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725346 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725470 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725537 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.725575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.725558 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshdg\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.826902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.826864 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.826986 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827027 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827111 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827138 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827186 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827320 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827365 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827433 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827465 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshdg\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.827673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.827642 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.828081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.828055 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.828239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.828218 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.828494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.828472 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.830564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.830537 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.830799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.830765 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.856816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.856786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshdg\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.857361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.857338 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.985094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.985007 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:22.991660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:22.991631 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:23.346011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.345983 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x"] Oct 29 22:44:23.346336 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:23.345983 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2892e72b_9eb9_4ffa_abfa_a45306280418.slice/crio-0e1f8afaf40dc311821c6cf2465a45819c4b3498bc3d3eaa6a9bc8b9af0d18dd WatchSource:0}: Error finding container 0e1f8afaf40dc311821c6cf2465a45819c4b3498bc3d3eaa6a9bc8b9af0d18dd: Status 404 returned error can't find the container with id 0e1f8afaf40dc311821c6cf2465a45819c4b3498bc3d3eaa6a9bc8b9af0d18dd Oct 29 22:44:23.349244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.349209 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:44:23.349362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.349286 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:44:23.349362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.349333 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:44:23.899541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.899505 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" event={"ID":"2892e72b-9eb9-4ffa-abfa-a45306280418","Type":"ContainerStarted","Data":"7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560"} Oct 29 22:44:23.899811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.899559 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" event={"ID":"2892e72b-9eb9-4ffa-abfa-a45306280418","Type":"ContainerStarted","Data":"0e1f8afaf40dc311821c6cf2465a45819c4b3498bc3d3eaa6a9bc8b9af0d18dd"} Oct 29 22:44:23.899957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.899936 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:23.900972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.900954 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:44:23.986096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:23.986068 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:24.037933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.037882 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podStartSLOduration=2.037864853 podStartE2EDuration="2.037864853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:23.980017243 +0000 UTC m=+1730.874092164" watchObservedRunningTime="2026-10-29 22:44:24.037864853 +0000 UTC m=+1730.931939775" Oct 29 22:44:24.375498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.374748 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:44:24.468547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.468522 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:44:24.505235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.505208 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.505387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.505260 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.505497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.505477 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.505665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.505645 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.505822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.505769 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.506356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.505967 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.506468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.506392 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.506468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.506463 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.506596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.506497 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.506596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.506567 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data\") pod \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\" (UID: \"31b187c6-0aa7-45eb-af5a-82f12ca47ae0\") " Oct 29 22:44:24.507147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.506325 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:24.507261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.507233 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.507332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.507299 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.507584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.507546 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.507954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.507928 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:24.508065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508048 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "kube-api-access-9gd6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.508163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508146 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.508221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508169 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.508221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508183 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.508221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508195 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.508221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508209 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.508340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508222 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.508480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.508458 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.509502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.509480 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token" pod "31b187c6-0aa7-45eb-af5a-82f12ca47ae0" (UID: "31b187c6-0aa7-45eb-af5a-82f12ca47ae0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.609157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609128 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.609351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609324 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.609483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609466 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.609581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609538 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.609581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609574 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.609694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609657 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.610071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609805 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.610071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609846 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.610071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.609886 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg\") pod \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\" (UID: \"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3\") " Oct 29 22:44:24.610071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.610071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610047 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.610354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610131 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.610354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610161 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:24.610354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610327 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.610904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610879 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610907 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610921 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610934 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gd6p\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-kube-api-access-9gd6p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610950 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610963 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610976 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.610990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.610989 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b187c6-0aa7-45eb-af5a-82f12ca47ae0-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.612084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.612058 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.612084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.612076 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.612231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.612218 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:24.612366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.612349 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg" pod "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" (UID: "acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3"). InnerVolumeSpecName "kube-api-access-8ktqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.713129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.713047 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.713129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.713079 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ktqg\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-kube-api-access-8ktqg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.713129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.713089 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.713129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.713100 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.914469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.914421 5367 generic.go:356] "Generic (PLEG): container finished" podID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerID="ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc" exitCode=0 Oct 29 22:44:24.914658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.914488 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" Oct 29 22:44:24.914658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.914515 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" event={"ID":"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3","Type":"ContainerDied","Data":"ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc"} Oct 29 22:44:24.914658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.914581 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4" event={"ID":"acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3","Type":"ContainerDied","Data":"bba25861367dd1bf748a83d7a6754ce7b638e687b721ed1e3a1f45adab70d866"} Oct 29 22:44:24.914658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.914613 5367 scope.go:122] "RemoveContainer" containerID="ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc" Oct 29 22:44:24.921127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.916745 5367 generic.go:356] "Generic (PLEG): container finished" podID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerID="e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac" exitCode=0 Oct 29 22:44:24.921127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.917776 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" Oct 29 22:44:24.921127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.918005 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" event={"ID":"31b187c6-0aa7-45eb-af5a-82f12ca47ae0","Type":"ContainerDied","Data":"e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac"} Oct 29 22:44:24.921127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.918054 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9" event={"ID":"31b187c6-0aa7-45eb-af5a-82f12ca47ae0","Type":"ContainerDied","Data":"9ee9976c63006c6725a60c2993f9188cd85c3409e6350c302e06566e7c018a25"} Oct 29 22:44:24.931903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.931887 5367 scope.go:122] "RemoveContainer" containerID="793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a" Oct 29 22:44:24.942663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.942646 5367 scope.go:122] "RemoveContainer" containerID="ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc" Oct 29 22:44:24.942941 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:24.942919 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc\": container with ID starting with ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc not found: ID does not exist" containerID="ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc" Oct 29 22:44:24.943030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.942947 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc"} err="failed to get container status \"ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc\": rpc error: code = NotFound desc = could not find container \"ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc\": container with ID starting with ec5db4cfc72b7007890a19b9e2395871013573f42b8ba92a82669e498b7addfc not found: ID does not exist" Oct 29 22:44:24.943030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.942963 5367 scope.go:122] "RemoveContainer" containerID="793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a" Oct 29 22:44:24.943199 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:24.943184 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a\": container with ID starting with 793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a not found: ID does not exist" containerID="793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a" Oct 29 22:44:24.943241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.943205 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a"} err="failed to get container status \"793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a\": rpc error: code = NotFound desc = could not find container \"793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a\": container with ID starting with 793b645249b0246079b28221898e8c012c9247e81a6a89c9ce5e206a031e767a not found: ID does not exist" Oct 29 22:44:24.943241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.943217 5367 scope.go:122] "RemoveContainer" containerID="e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac" Oct 29 22:44:24.953486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.953469 5367 scope.go:122] "RemoveContainer" containerID="e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac" Oct 29 22:44:24.953775 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:24.953753 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac\": container with ID starting with e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac not found: ID does not exist" containerID="e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac" Oct 29 22:44:24.953858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.953780 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac"} err="failed to get container status \"e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac\": rpc error: code = NotFound desc = could not find container \"e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac\": container with ID starting with e22ceb07bc2e05e31cf5fb6ee339ab2d90b9100c8cb5539dfbc56b94cee62cac not found: ID does not exist" Oct 29 22:44:24.980795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.980740 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9"] Oct 29 22:44:24.992156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.992136 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:24.993332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:24.993313 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-p9rx9"] Oct 29 22:44:25.019375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.019348 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4"] Oct 29 22:44:25.024014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.023993 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-xvvt4"] Oct 29 22:44:25.230041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.230012 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" path="/var/lib/kubelet/pods/31b187c6-0aa7-45eb-af5a-82f12ca47ae0/volumes" Oct 29 22:44:25.230787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.230765 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" path="/var/lib/kubelet/pods/acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3/volumes" Oct 29 22:44:25.487872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.487851 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:44:25.630131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630057 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630134 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630232 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630279 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630315 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630389 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630464 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630657 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630723 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf\") pod \"6be234cd-119e-42c7-918d-03b3259114b4\" (UID: \"6be234cd-119e-42c7-918d-03b3259114b4\") " Oct 29 22:44:25.630952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630458 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:25.630952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630776 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:25.630952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.630946 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:25.631404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631376 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:25.631594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631574 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:25.631922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631903 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.631922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631922 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.632104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631931 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.632104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631940 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.632104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.631948 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be234cd-119e-42c7-918d-03b3259114b4-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.632530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.632507 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:25.632726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.632687 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:25.633097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.633070 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:25.633773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.633753 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf" pod "6be234cd-119e-42c7-918d-03b3259114b4" (UID: "6be234cd-119e-42c7-918d-03b3259114b4"). InnerVolumeSpecName "kube-api-access-pbhbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:25.733394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.733360 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be234cd-119e-42c7-918d-03b3259114b4-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.733394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.733393 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbhbf\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-kube-api-access-pbhbf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.733582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.733405 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be234cd-119e-42c7-918d-03b3259114b4-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.733582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.733417 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be234cd-119e-42c7-918d-03b3259114b4-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.906098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.899991 5367 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:51142->10.133.0.104:15021: read: connection reset by peer" start-of-body= Oct 29 22:44:25.906098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.900055 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:51142->10.133.0.104:15021: read: connection reset by peer" Oct 29 22:44:25.930167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.930125 5367 generic.go:356] "Generic (PLEG): container finished" podID="6be234cd-119e-42c7-918d-03b3259114b4" containerID="9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970" exitCode=0 Oct 29 22:44:25.930309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.930190 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" Oct 29 22:44:25.930309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.930187 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" event={"ID":"6be234cd-119e-42c7-918d-03b3259114b4","Type":"ContainerDied","Data":"9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970"} Oct 29 22:44:25.930504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.930340 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp" event={"ID":"6be234cd-119e-42c7-918d-03b3259114b4","Type":"ContainerDied","Data":"f4a76d8d9d0d7c8aff2e4be524544cb188eb7de32958c9a0e283c8cdd49f3b90"} Oct 29 22:44:25.930504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.930372 5367 scope.go:122] "RemoveContainer" containerID="9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970" Oct 29 22:44:25.932317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.932291 5367 generic.go:356] "Generic (PLEG): container finished" podID="1635df12-4155-49bd-858b-3b4674a893c7" containerID="c6482d182bf5ce9a3fa93defa69b0e36d23a33a9ae8270e5f53a06950b797f30" exitCode=0 Oct 29 22:44:25.932317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.932378 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" event={"ID":"1635df12-4155-49bd-858b-3b4674a893c7","Type":"ContainerDied","Data":"c6482d182bf5ce9a3fa93defa69b0e36d23a33a9ae8270e5f53a06950b797f30"} Oct 29 22:44:25.934309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.934287 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:25.935294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.935278 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:44:25.957951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.957920 5367 scope.go:122] "RemoveContainer" containerID="7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c" Oct 29 22:44:25.973890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.973635 5367 scope.go:122] "RemoveContainer" containerID="9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970" Oct 29 22:44:25.974350 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:25.974199 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970\": container with ID starting with 9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970 not found: ID does not exist" containerID="9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970" Oct 29 22:44:25.974350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.974236 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970"} err="failed to get container status \"9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970\": rpc error: code = NotFound desc = could not find container \"9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970\": container with ID starting with 9eb22873813f527a5a6dc75ec1dcf5ebc03336f4ebf77c06e69eba8bedceb970 not found: ID does not exist" Oct 29 22:44:25.974350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.974264 5367 scope.go:122] "RemoveContainer" containerID="7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c" Oct 29 22:44:25.974776 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:25.974674 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c\": container with ID starting with 7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c not found: ID does not exist" containerID="7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c" Oct 29 22:44:25.974776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:25.974738 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c"} err="failed to get container status \"7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c\": rpc error: code = NotFound desc = could not find container \"7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c\": container with ID starting with 7247d389bcd79630093f9303e619880a4f8b3d96b9db189395360965bad2d40c not found: ID does not exist" Oct 29 22:44:26.137193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.136018 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp"] Oct 29 22:44:26.144721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.143152 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c06e05-openshift-default-cfb8d84fc-zb8gp"] Oct 29 22:44:26.241311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.241281 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.350732 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.350807 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.350973 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.351004 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.351032 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.351059 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.351100 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.351121 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.351222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.351159 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq\") pod \"1635df12-4155-49bd-858b-3b4674a893c7\" (UID: \"1635df12-4155-49bd-858b-3b4674a893c7\") " Oct 29 22:44:26.352680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.352648 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:26.352926 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.352899 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.353016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.352996 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.353073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.353013 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.353206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.353187 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.354606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.354579 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "kube-api-access-k8qtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:26.356027 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.356004 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:26.356367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.356343 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:26.357110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.357088 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy" pod "1635df12-4155-49bd-858b-3b4674a893c7" (UID: "1635df12-4155-49bd-858b-3b4674a893c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.410493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.410305 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:44:26.452958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452878 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1635df12-4155-49bd-858b-3b4674a893c7-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.452958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452909 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.452958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452923 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.452958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452935 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.452958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452948 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1635df12-4155-49bd-858b-3b4674a893c7-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.453261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452960 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qtq\" (UniqueName: \"kubernetes.io/projected/1635df12-4155-49bd-858b-3b4674a893c7-kube-api-access-k8qtq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.453261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.452977 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.453261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.453011 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.453261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.453019 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1635df12-4155-49bd-858b-3b4674a893c7-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.553600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.553546 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.553809 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.553921 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554025 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554089 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554093 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.554222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554198 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554227 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554339 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token\") pod \"0fe5c579-0c5f-4404-be47-1c61db827670\" (UID: \"0fe5c579-0c5f-4404-be47-1c61db827670\") " Oct 29 22:44:26.554437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554405 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.555760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554623 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:26.555760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554769 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.555760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.554871 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.555760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.555424 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.555760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.555761 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe5c579-0c5f-4404-be47-1c61db827670-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.556088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.555778 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.556088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.555790 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.556088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.555804 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.556334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.556312 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "kube-api-access-v2vrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:26.556991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.556969 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.557257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.557234 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:26.557331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.557301 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo" pod "0fe5c579-0c5f-4404-be47-1c61db827670" (UID: "0fe5c579-0c5f-4404-be47-1c61db827670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:26.657229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.657183 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe5c579-0c5f-4404-be47-1c61db827670-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.657229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.657227 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe5c579-0c5f-4404-be47-1c61db827670-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.657229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.657239 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.657440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.657252 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vrc\" (UniqueName: \"kubernetes.io/projected/0fe5c579-0c5f-4404-be47-1c61db827670-kube-api-access-v2vrc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.698068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.698027 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt"] Oct 29 22:44:26.699357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699330 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" Oct 29 22:44:26.699357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699355 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699368 5367 cpu_manager.go:414] "containerMap: removing container" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699376 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699439 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699446 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699473 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699480 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699505 5367 cpu_manager.go:414] "containerMap: removing container" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" Oct 29 22:44:26.699521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699512 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699535 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699543 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699741 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699756 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699769 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699787 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" Oct 29 22:44:26.699806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699800 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" containerName="istio-proxy" Oct 29 22:44:26.700028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699812 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b187c6-0aa7-45eb-af5a-82f12ca47ae0" containerName="istio-proxy" Oct 29 22:44:26.700028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.699823 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1635df12-4155-49bd-858b-3b4674a893c7" containerName="istio-proxy" Oct 29 22:44:26.706466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.706409 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:26.722137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.722113 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6bc7b2-dockercfg-w47qb\"" Oct 29 22:44:26.722269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.722167 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6bc7b2\"" Oct 29 22:44:26.724050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.724027 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt"] Oct 29 22:44:26.763537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.763506 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74"] Oct 29 22:44:26.764398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.764379 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" Oct 29 22:44:26.764449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.764400 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" Oct 29 22:44:26.764449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.764412 5367 cpu_manager.go:414] "containerMap: removing container" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" Oct 29 22:44:26.764449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.764417 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb7a0b4-b57d-4d5d-b7c7-ff89ccc5f3a3" containerName="istio-proxy" Oct 29 22:44:26.764549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.764541 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be234cd-119e-42c7-918d-03b3259114b4" containerName="istio-proxy" Oct 29 22:44:26.768506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.768483 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:26.791609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.791582 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74"] Oct 29 22:44:26.872279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.871633 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:26.923149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.923105 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqg7\" (UniqueName: \"kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7\") pod \"payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:26.930087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.930056 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd8q\" (UniqueName: \"kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q\") pod \"payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:26.937721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:26.930482 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:27.051626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.050386 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:27.051626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.051282 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:27.070884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.070854 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqg7\" (UniqueName: \"kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7\") pod \"payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:27.071295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.071277 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd8q\" (UniqueName: \"kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q\") pod \"payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:27.071537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.071523 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:27.071747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.071719 5367 generic.go:356] "Generic (PLEG): container finished" podID="0fe5c579-0c5f-4404-be47-1c61db827670" containerID="4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841" exitCode=0 Oct 29 22:44:27.072477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.072452 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" Oct 29 22:44:27.072864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.072788 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" event={"ID":"0fe5c579-0c5f-4404-be47-1c61db827670","Type":"ContainerDied","Data":"4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841"} Oct 29 22:44:27.072864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.072847 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk" event={"ID":"0fe5c579-0c5f-4404-be47-1c61db827670","Type":"ContainerDied","Data":"15f5df4faafd6deff9b354fff6ad434389b1692fe44d4aec5030ecf8bfbb6988"} Oct 29 22:44:27.072978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.072875 5367 scope.go:122] "RemoveContainer" containerID="4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841" Oct 29 22:44:27.073030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.072806 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:27.080176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.075470 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" Oct 29 22:44:27.080346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.080295 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt" event={"ID":"1635df12-4155-49bd-858b-3b4674a893c7","Type":"ContainerDied","Data":"33c4d22cd29748900a91d61e2312fb88627c45fbc93006968e7a218768c5016e"} Oct 29 22:44:27.126358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.125265 5367 scope.go:122] "RemoveContainer" containerID="4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f" Oct 29 22:44:27.152387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.140016 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd8q\" (UniqueName: \"kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q\") pod \"payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:27.219021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.197445 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqg7\" (UniqueName: \"kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7\") pod \"payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:27.307990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.307965 5367 scope.go:122] "RemoveContainer" containerID="4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841" Oct 29 22:44:27.313711 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:27.313631 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841\": container with ID starting with 4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841 not found: ID does not exist" containerID="4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841" Oct 29 22:44:27.314689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.314632 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841"} err="failed to get container status \"4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841\": rpc error: code = NotFound desc = could not find container \"4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841\": container with ID starting with 4f0bd415fe40c08faa47a289cb1061e58ff248de43a415148fd6e6318328f841 not found: ID does not exist" Oct 29 22:44:27.314689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.314692 5367 scope.go:122] "RemoveContainer" containerID="4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f" Oct 29 22:44:27.316902 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:27.315132 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f\": container with ID starting with 4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f not found: ID does not exist" containerID="4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f" Oct 29 22:44:27.316902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.315180 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f"} err="failed to get container status \"4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f\": rpc error: code = NotFound desc = could not find container \"4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f\": container with ID starting with 4485cb012c07f858eda53267b5e29cadd9ff807ed104fc190861b9783f1f263f not found: ID does not exist" Oct 29 22:44:27.316902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.315202 5367 scope.go:122] "RemoveContainer" containerID="c6482d182bf5ce9a3fa93defa69b0e36d23a33a9ae8270e5f53a06950b797f30" Oct 29 22:44:27.332849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.322084 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be234cd-119e-42c7-918d-03b3259114b4" path="/var/lib/kubelet/pods/6be234cd-119e-42c7-918d-03b3259114b4/volumes" Oct 29 22:44:27.332849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.322493 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:27.332849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.323158 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt"] Oct 29 22:44:27.332849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.323194 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-fqmpt"] Oct 29 22:44:27.332849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.329380 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk"] Oct 29 22:44:27.338827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.336767 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-djcwk"] Oct 29 22:44:27.389169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.389131 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:27.890813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.890781 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt"] Oct 29 22:44:27.893195 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:27.893162 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae5e8d4_de0b_4ecb_ae78_7495e9586400.slice/crio-356cef412047c0b5ac243501f44934164358e4df895674b286bee3c291dddc5e WatchSource:0}: Error finding container 356cef412047c0b5ac243501f44934164358e4df895674b286bee3c291dddc5e: Status 404 returned error can't find the container with id 356cef412047c0b5ac243501f44934164358e4df895674b286bee3c291dddc5e Oct 29 22:44:27.990474 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:27.990446 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6b42d6_f6ad_4962_8a4c_2b49f4e101b9.slice/crio-c6e7f0765fff418a4f2f611f23fab9594955e76097474f4816c8ae2519a65464 WatchSource:0}: Error finding container c6e7f0765fff418a4f2f611f23fab9594955e76097474f4816c8ae2519a65464: Status 404 returned error can't find the container with id c6e7f0765fff418a4f2f611f23fab9594955e76097474f4816c8ae2519a65464 Oct 29 22:44:27.990570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:27.990540 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74"] Oct 29 22:44:28.086132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:28.086104 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" event={"ID":"fae5e8d4-de0b-4ecb-ae78-7495e9586400","Type":"ContainerStarted","Data":"9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b"} Oct 29 22:44:28.086256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:28.086154 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" event={"ID":"fae5e8d4-de0b-4ecb-ae78-7495e9586400","Type":"ContainerStarted","Data":"356cef412047c0b5ac243501f44934164358e4df895674b286bee3c291dddc5e"} Oct 29 22:44:28.086256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:28.086200 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:28.089423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:28.089394 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" event={"ID":"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9","Type":"ContainerStarted","Data":"c6e7f0765fff418a4f2f611f23fab9594955e76097474f4816c8ae2519a65464"} Oct 29 22:44:28.137985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:28.137929 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" podStartSLOduration=2.137915083 podStartE2EDuration="2.137915083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:28.132285368 +0000 UTC m=+1735.026360288" watchObservedRunningTime="2026-10-29 22:44:28.137915083 +0000 UTC m=+1735.031990003" Oct 29 22:44:28.995540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:28.995489 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd"] Oct 29 22:44:29.000692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.000668 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.020741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.019821 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-172c10-dockercfg-9kzhl\"" Oct 29 22:44:29.021310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.021280 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd"] Oct 29 22:44:29.027046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.027020 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-172c10\"" Oct 29 22:44:29.095723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.088968 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9nw\" (UniqueName: \"kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw\") pod \"payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.095723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.089057 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.214762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.205321 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9nw\" (UniqueName: \"kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw\") pod \"payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.215095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.215064 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.217396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.217371 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.502866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.420658 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9nw\" (UniqueName: \"kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw\") pod \"payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.528719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.528636 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" podStartSLOduration=3.528617837 podStartE2EDuration="3.528617837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:29.420971003 +0000 UTC m=+1736.315045925" watchObservedRunningTime="2026-10-29 22:44:29.528617837 +0000 UTC m=+1736.422692758" Oct 29 22:44:29.619731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.619062 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:29.660786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.660733 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe5c579-0c5f-4404-be47-1c61db827670" path="/var/lib/kubelet/pods/0fe5c579-0c5f-4404-be47-1c61db827670/volumes" Oct 29 22:44:29.668729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.661586 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1635df12-4155-49bd-858b-3b4674a893c7" path="/var/lib/kubelet/pods/1635df12-4155-49bd-858b-3b4674a893c7/volumes" Oct 29 22:44:29.696122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.696085 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h"] Oct 29 22:44:29.815280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.799298 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" event={"ID":"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9","Type":"ContainerStarted","Data":"4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9"} Oct 29 22:44:29.815280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.799363 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:29.815280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.799395 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h"] Oct 29 22:44:29.815280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:29.799509 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.021298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.017355 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-172c10-57c45696d8-9697h\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.021298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.017406 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqnv\" (UniqueName: \"kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv\") pod \"payload-processing-e2e-inf-a-172c10-57c45696d8-9697h\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.126674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.126573 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-172c10-57c45696d8-9697h\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.126959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.126931 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqnv\" (UniqueName: \"kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv\") pod \"payload-processing-e2e-inf-a-172c10-57c45696d8-9697h\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.149719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.138822 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-172c10-57c45696d8-9697h\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.212959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.212911 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.214534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.214502 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:44:30.290722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.278390 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqnv\" (UniqueName: \"kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv\") pod \"payload-processing-e2e-inf-a-172c10-57c45696d8-9697h\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.292772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.289470 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.304730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.294992 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:30.457780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.457743 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.773069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.772927 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252"] Oct 29 22:44:30.870746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.870712 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ae775-l4252"] Oct 29 22:44:30.974639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:30.974604 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd"] Oct 29 22:44:30.976238 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:30.976212 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1646498f_db1a_4e03_8701_7c02e75d1f33.slice/crio-19308a26a7489ddbb0d84ee272cc460ab70bbb524209b450020bbe5a28c7d874 WatchSource:0}: Error finding container 19308a26a7489ddbb0d84ee272cc460ab70bbb524209b450020bbe5a28c7d874: Status 404 returned error can't find the container with id 19308a26a7489ddbb0d84ee272cc460ab70bbb524209b450020bbe5a28c7d874 Oct 29 22:44:31.083026 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:31.082996 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677161a8_a4f8_4c50_93aa_3a73b8818297.slice/crio-97d53e7fda3c5a78330266083d65587fe613fde554a02351653d60f6edc7f5fb WatchSource:0}: Error finding container 97d53e7fda3c5a78330266083d65587fe613fde554a02351653d60f6edc7f5fb: Status 404 returned error can't find the container with id 97d53e7fda3c5a78330266083d65587fe613fde554a02351653d60f6edc7f5fb Oct 29 22:44:31.083495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.083472 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h"] Oct 29 22:44:31.227171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.227138 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2262a60-83be-4f31-946e-aa4135a45e37" path="/var/lib/kubelet/pods/e2262a60-83be-4f31-946e-aa4135a45e37/volumes" Oct 29 22:44:31.486640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.486433 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" event={"ID":"1646498f-db1a-4e03-8701-7c02e75d1f33","Type":"ContainerStarted","Data":"4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146"} Oct 29 22:44:31.486640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.486564 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:31.486640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.486596 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" event={"ID":"1646498f-db1a-4e03-8701-7c02e75d1f33","Type":"ContainerStarted","Data":"19308a26a7489ddbb0d84ee272cc460ab70bbb524209b450020bbe5a28c7d874"} Oct 29 22:44:31.488884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.488859 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" event={"ID":"677161a8-a4f8-4c50-93aa-3a73b8818297","Type":"ContainerStarted","Data":"d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9"} Oct 29 22:44:31.489094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.489076 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:31.489189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.489113 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" event={"ID":"677161a8-a4f8-4c50-93aa-3a73b8818297","Type":"ContainerStarted","Data":"97d53e7fda3c5a78330266083d65587fe613fde554a02351653d60f6edc7f5fb"} Oct 29 22:44:31.588931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.588882 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" podStartSLOduration=2.588867221 podStartE2EDuration="2.588867221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:31.587056419 +0000 UTC m=+1738.481131338" watchObservedRunningTime="2026-10-29 22:44:31.588867221 +0000 UTC m=+1738.482942164" Oct 29 22:44:31.592433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:31.592393 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" podStartSLOduration=3.592381297 podStartE2EDuration="3.592381297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:31.539908315 +0000 UTC m=+1738.433983245" watchObservedRunningTime="2026-10-29 22:44:31.592381297 +0000 UTC m=+1738.486456217" Oct 29 22:44:34.268191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.268156 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4"] Oct 29 22:44:34.274478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.274459 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.295429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.295406 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4"] Oct 29 22:44:34.301338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.301315 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:44:34.301477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.301451 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:44:34.392667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.392633 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.392880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.392862 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.393025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.393007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ld7\" (UniqueName: \"kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.494760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.494694 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.494918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.494832 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.494918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.494898 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ld7\" (UniqueName: \"kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.495401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.495377 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.497181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.497161 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.515096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.515066 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ld7\" (UniqueName: \"kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7\") pod \"maas-api-revoke-sub-dc47886c-vdwp4\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.590120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.590097 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:34.893368 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:34.893335 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3305448f_5002_4224_afb8_8b1c50294f21.slice/crio-edf55c640377c05c8e24ed4ea8e70a524c30ad2637630dea5deeb634cdd2d2a1 WatchSource:0}: Error finding container edf55c640377c05c8e24ed4ea8e70a524c30ad2637630dea5deeb634cdd2d2a1: Status 404 returned error can't find the container with id edf55c640377c05c8e24ed4ea8e70a524c30ad2637630dea5deeb634cdd2d2a1 Oct 29 22:44:34.894256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:34.894230 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4"] Oct 29 22:44:35.526544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:35.526458 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" event={"ID":"3305448f-5002-4224-afb8-8b1c50294f21","Type":"ContainerStarted","Data":"dd7354b141fdd5f3434c876147b4394dfdc534fba6326293e4171e6e1ce0dcd1"} Oct 29 22:44:35.526544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:35.526515 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" event={"ID":"3305448f-5002-4224-afb8-8b1c50294f21","Type":"ContainerStarted","Data":"edf55c640377c05c8e24ed4ea8e70a524c30ad2637630dea5deeb634cdd2d2a1"} Oct 29 22:44:35.569427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:35.569367 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" podStartSLOduration=1.5693494609999998 podStartE2EDuration="1.569349461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:35.565716761 +0000 UTC m=+1742.459791680" watchObservedRunningTime="2026-10-29 22:44:35.569349461 +0000 UTC m=+1742.463424382" Oct 29 22:44:36.538525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:36.538490 5367 generic.go:356] "Generic (PLEG): container finished" podID="3305448f-5002-4224-afb8-8b1c50294f21" containerID="dd7354b141fdd5f3434c876147b4394dfdc534fba6326293e4171e6e1ce0dcd1" exitCode=0 Oct 29 22:44:36.539009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:36.538535 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" event={"ID":"3305448f-5002-4224-afb8-8b1c50294f21","Type":"ContainerDied","Data":"dd7354b141fdd5f3434c876147b4394dfdc534fba6326293e4171e6e1ce0dcd1"} Oct 29 22:44:37.781904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:37.781878 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:38.073558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.067039 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca\") pod \"3305448f-5002-4224-afb8-8b1c50294f21\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " Oct 29 22:44:38.073558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.067234 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7\" (UniqueName: \"kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7\") pod \"3305448f-5002-4224-afb8-8b1c50294f21\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " Oct 29 22:44:38.073558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.067394 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca\") pod \"3305448f-5002-4224-afb8-8b1c50294f21\" (UID: \"3305448f-5002-4224-afb8-8b1c50294f21\") " Oct 29 22:44:38.073558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.067534 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca" pod "3305448f-5002-4224-afb8-8b1c50294f21" (UID: "3305448f-5002-4224-afb8-8b1c50294f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:38.077514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.076862 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:38.101897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.096929 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7" pod "3305448f-5002-4224-afb8-8b1c50294f21" (UID: "3305448f-5002-4224-afb8-8b1c50294f21"). InnerVolumeSpecName "kube-api-access-j8ld7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:38.116722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.102249 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca" pod "3305448f-5002-4224-afb8-8b1c50294f21" (UID: "3305448f-5002-4224-afb8-8b1c50294f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:38.178941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.178906 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ld7\" (UniqueName: \"kubernetes.io/projected/3305448f-5002-4224-afb8-8b1c50294f21-kube-api-access-j8ld7\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:38.178941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.178936 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3305448f-5002-4224-afb8-8b1c50294f21-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:38.236677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.236649 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7"] Oct 29 22:44:38.237520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.237504 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3305448f-5002-4224-afb8-8b1c50294f21" containerName="revoke-keys" Oct 29 22:44:38.237564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.237522 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3305448f-5002-4224-afb8-8b1c50294f21" containerName="revoke-keys" Oct 29 22:44:38.237640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.237630 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3305448f-5002-4224-afb8-8b1c50294f21" containerName="revoke-keys" Oct 29 22:44:38.241934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.241915 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.260659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.260634 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7"] Oct 29 22:44:38.283230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.283195 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.283566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.283543 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ppgg\" (UniqueName: \"kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.283640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.283625 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.385079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.384994 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ppgg\" (UniqueName: \"kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.385079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.385057 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.385329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.385164 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.385779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.385754 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.387464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.387447 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.405783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.405760 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ppgg\" (UniqueName: \"kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg\") pod \"maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.558241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.558208 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:38.559640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.559612 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" event={"ID":"3305448f-5002-4224-afb8-8b1c50294f21","Type":"ContainerDied","Data":"edf55c640377c05c8e24ed4ea8e70a524c30ad2637630dea5deeb634cdd2d2a1"} Oct 29 22:44:38.559780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.559646 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4" Oct 29 22:44:38.559780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.559663 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf55c640377c05c8e24ed4ea8e70a524c30ad2637630dea5deeb634cdd2d2a1" Oct 29 22:44:38.860508 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:44:38.860476 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81bdf6c_42eb_40d3_8d1f_cc2976e61529.slice/crio-679f93b2f6f88954b93be9865b399ed8e764383fc27d68b736ba41ebf194174e WatchSource:0}: Error finding container 679f93b2f6f88954b93be9865b399ed8e764383fc27d68b736ba41ebf194174e: Status 404 returned error can't find the container with id 679f93b2f6f88954b93be9865b399ed8e764383fc27d68b736ba41ebf194174e Oct 29 22:44:38.861012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:38.860984 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7"] Oct 29 22:44:39.370637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:39.370602 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:44:39.569986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:39.569944 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" event={"ID":"c81bdf6c-42eb-40d3-8d1f-cc2976e61529","Type":"ContainerStarted","Data":"9c041f0c61642b54b01234c127eb66685063bc7566b26855f3820d66e861f502"} Oct 29 22:44:39.570142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:39.570010 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" event={"ID":"c81bdf6c-42eb-40d3-8d1f-cc2976e61529","Type":"ContainerStarted","Data":"679f93b2f6f88954b93be9865b399ed8e764383fc27d68b736ba41ebf194174e"} Oct 29 22:44:39.612750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:39.612691 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" podStartSLOduration=1.612678597 podStartE2EDuration="1.612678597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:39.610205113 +0000 UTC m=+1746.504280029" watchObservedRunningTime="2026-10-29 22:44:39.612678597 +0000 UTC m=+1746.506753516" Oct 29 22:44:40.581400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:40.581366 5367 generic.go:356] "Generic (PLEG): container finished" podID="c81bdf6c-42eb-40d3-8d1f-cc2976e61529" containerID="9c041f0c61642b54b01234c127eb66685063bc7566b26855f3820d66e861f502" exitCode=0 Oct 29 22:44:40.581807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:40.581455 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" event={"ID":"c81bdf6c-42eb-40d3-8d1f-cc2976e61529","Type":"ContainerDied","Data":"9c041f0c61642b54b01234c127eb66685063bc7566b26855f3820d66e861f502"} Oct 29 22:44:41.492110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.492081 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:44:41.838316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.838291 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:41.941571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.940445 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg\" (UniqueName: \"kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg\") pod \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " Oct 29 22:44:41.941571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.940614 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca\") pod \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " Oct 29 22:44:41.941571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.940752 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca\") pod \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\" (UID: \"c81bdf6c-42eb-40d3-8d1f-cc2976e61529\") " Oct 29 22:44:41.941571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.941465 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca" pod "c81bdf6c-42eb-40d3-8d1f-cc2976e61529" (UID: "c81bdf6c-42eb-40d3-8d1f-cc2976e61529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:41.944192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.944164 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg" pod "c81bdf6c-42eb-40d3-8d1f-cc2976e61529" (UID: "c81bdf6c-42eb-40d3-8d1f-cc2976e61529"). InnerVolumeSpecName "kube-api-access-7ppgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:41.944582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:41.944550 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca" pod "c81bdf6c-42eb-40d3-8d1f-cc2976e61529" (UID: "c81bdf6c-42eb-40d3-8d1f-cc2976e61529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:42.043111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.043071 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:42.043111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.043110 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:42.043307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.043120 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ppgg\" (UniqueName: \"kubernetes.io/projected/c81bdf6c-42eb-40d3-8d1f-cc2976e61529-kube-api-access-7ppgg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:42.500875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.500848 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:44:42.501070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.500914 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:44:42.621035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.621007 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" Oct 29 22:44:42.621182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.621032 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7" event={"ID":"c81bdf6c-42eb-40d3-8d1f-cc2976e61529","Type":"ContainerDied","Data":"679f93b2f6f88954b93be9865b399ed8e764383fc27d68b736ba41ebf194174e"} Oct 29 22:44:42.621182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:42.621092 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679f93b2f6f88954b93be9865b399ed8e764383fc27d68b736ba41ebf194174e" Oct 29 22:44:50.649931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:50.649897 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789"] Oct 29 22:44:50.650434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:50.650102 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" containerID="cri-o://4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8" gracePeriod=30 Oct 29 22:44:52.333216 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:52.333183 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:44:55.939051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.938987 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:44:55.944621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.944592 5367 generic.go:356] "Generic (PLEG): container finished" podID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerID="4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8" exitCode=0 Oct 29 22:44:55.944743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.944636 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" Oct 29 22:44:55.944743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.944673 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" event={"ID":"86451d4b-a607-4d8e-90ef-ab2267aa7705","Type":"ContainerDied","Data":"4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8"} Oct 29 22:44:55.944835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.944771 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789" event={"ID":"86451d4b-a607-4d8e-90ef-ab2267aa7705","Type":"ContainerDied","Data":"70f2d991c263aac9e13f013e881a2e139f65209dbe37f2a75d96ffb7d0931f32"} Oct 29 22:44:55.944835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.944797 5367 scope.go:122] "RemoveContainer" containerID="4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8" Oct 29 22:44:55.956399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.956380 5367 scope.go:122] "RemoveContainer" containerID="19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234" Oct 29 22:44:55.967685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.967669 5367 scope.go:122] "RemoveContainer" containerID="4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8" Oct 29 22:44:55.968056 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:55.968024 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8\": container with ID starting with 4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8 not found: ID does not exist" containerID="4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8" Oct 29 22:44:55.968147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.968062 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8"} err="failed to get container status \"4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8\": rpc error: code = NotFound desc = could not find container \"4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8\": container with ID starting with 4569c3b06cd0bb9208e10df775a8f3f58a38e59a9b9864eb580fa015c7c047f8 not found: ID does not exist" Oct 29 22:44:55.968147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.968078 5367 scope.go:122] "RemoveContainer" containerID="19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234" Oct 29 22:44:55.968349 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:44:55.968324 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234\": container with ID starting with 19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234 not found: ID does not exist" containerID="19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234" Oct 29 22:44:55.968395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.968359 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234"} err="failed to get container status \"19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234\": rpc error: code = NotFound desc = could not find container \"19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234\": container with ID starting with 19596c6eaf9fa15f36b497fd6adfbfa00c3037e74152040068f2a41dfb461234 not found: ID does not exist" Oct 29 22:44:55.973130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973106 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973170 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973275 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973380 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973419 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973436 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:55.973563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973472 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:55.973563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973487 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973617 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973649 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.973806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973648 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:55.973806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973724 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:55.973806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973749 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55\") pod \"86451d4b-a607-4d8e-90ef-ab2267aa7705\" (UID: \"86451d4b-a607-4d8e-90ef-ab2267aa7705\") " Oct 29 22:44:55.974035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.973904 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:55.974870 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.974844 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.975004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.974871 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.975004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.974885 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.975004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.974901 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86451d4b-a607-4d8e-90ef-ab2267aa7705-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.975004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.974915 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.975291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.975272 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:55.975687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.975670 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:55.975800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.975755 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:55.975848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:55.975836 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55" pod "86451d4b-a607-4d8e-90ef-ab2267aa7705" (UID: "86451d4b-a607-4d8e-90ef-ab2267aa7705"). InnerVolumeSpecName "kube-api-access-qgp55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:56.076824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:56.076785 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:56.076824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:56.076824 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:56.077042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:56.076836 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgp55\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-kube-api-access-qgp55\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:56.077042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:56.076850 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86451d4b-a607-4d8e-90ef-ab2267aa7705-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:44:56.307936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:56.307901 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789"] Oct 29 22:44:56.317881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:56.317852 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-w8789"] Oct 29 22:44:57.226311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:44:57.226280 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" path="/var/lib/kubelet/pods/86451d4b-a607-4d8e-90ef-ab2267aa7705/volumes" Oct 29 22:45:00.152206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.152168 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc"] Oct 29 22:45:00.153029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153012 5367 cpu_manager.go:414] "containerMap: removing container" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" Oct 29 22:45:00.153029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153029 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" Oct 29 22:45:00.153128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153049 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c81bdf6c-42eb-40d3-8d1f-cc2976e61529" containerName="revoke-keys" Oct 29 22:45:00.153128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153054 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81bdf6c-42eb-40d3-8d1f-cc2976e61529" containerName="revoke-keys" Oct 29 22:45:00.153128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153067 5367 cpu_manager.go:414] "containerMap: removing container" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" Oct 29 22:45:00.153128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153072 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" Oct 29 22:45:00.153283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153217 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" Oct 29 22:45:00.153283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153227 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81bdf6c-42eb-40d3-8d1f-cc2976e61529" containerName="revoke-keys" Oct 29 22:45:00.153283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.153235 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="86451d4b-a607-4d8e-90ef-ab2267aa7705" containerName="istio-proxy" Oct 29 22:45:00.158922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.158902 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.160785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.160763 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487"] Oct 29 22:45:00.165617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.165594 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.166288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.166265 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd"] Oct 29 22:45:00.171836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.171816 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.171836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.171825 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x"] Oct 29 22:45:00.176623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.176605 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.183039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.183018 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc"] Oct 29 22:45:00.211551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.193236 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:45:00.211551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.211301 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487"] Oct 29 22:45:00.223110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.222910 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x"] Oct 29 22:45:00.231450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.231421 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd"] Oct 29 22:45:00.237585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.237559 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d"] Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.243449 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.244227 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6"] Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245387 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245450 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngzr\" (UniqueName: \"kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245544 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245578 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245777 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76phr\" (UniqueName: \"kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr\") pod \"maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245918 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.245960 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmggf\" (UniqueName: \"kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf\") pod \"maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.250005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.246017 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvhp\" (UniqueName: \"kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.252322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.252300 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.257293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.257274 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp"] Oct 29 22:45:00.270068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.268763 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d"] Oct 29 22:45:00.270068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.268906 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.272486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.270738 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6"] Oct 29 22:45:00.277217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.276318 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5"] Oct 29 22:45:00.283925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.283194 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq"] Oct 29 22:45:00.283925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.283300 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.289968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.289777 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.319140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.318870 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp"] Oct 29 22:45:00.326924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.326872 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5"] Oct 29 22:45:00.335328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.335273 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq"] Oct 29 22:45:00.343345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.342998 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h"] Oct 29 22:45:00.347275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.347245 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.349812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.349782 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.349953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.349831 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.349953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.349910 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnzx\" (UniqueName: \"kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.349953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.349944 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.350124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.349980 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.350124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350031 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvv97\" (UniqueName: \"kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.350124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350069 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdw8\" (UniqueName: \"kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.350305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350150 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76phr\" (UniqueName: \"kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr\") pod \"maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.350305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350179 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.350305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350230 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjqk\" (UniqueName: \"kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.350305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350287 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.350554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350337 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.350554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350378 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmggf\" (UniqueName: \"kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf\") pod \"maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.350554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350458 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvhp\" (UniqueName: \"kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.350672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350560 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5fb\" (UniqueName: \"kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.350672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350641 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.350820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.350722 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngzr\" (UniqueName: \"kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.351804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.351778 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.352468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.352443 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.353146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.353121 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.353807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.353788 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.360039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.360018 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h"] Oct 29 22:45:00.360134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.360122 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.380799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.380775 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngzr\" (UniqueName: \"kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.384497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.384032 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76phr\" (UniqueName: \"kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr\") pod \"maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.384879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.384852 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmggf\" (UniqueName: \"kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf\") pod \"maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.385588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.385529 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvhp\" (UniqueName: \"kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.452731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452619 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnzx\" (UniqueName: \"kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.452731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452657 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.452731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452680 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.452957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452755 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvv97\" (UniqueName: \"kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.452957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452802 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdw8\" (UniqueName: \"kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.452957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452875 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.452957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452925 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjqk\" (UniqueName: \"kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.453167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.452965 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.453167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453102 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.453167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453152 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5fb\" (UniqueName: \"kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.453365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453324 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcz7\" (UniqueName: \"kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7\") pod \"maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.453560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453538 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.453619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453571 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.453619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453599 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.453787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453766 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.453958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.453942 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.454115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.454097 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.473521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.473494 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnzx\" (UniqueName: \"kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.473632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.473537 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5fb\" (UniqueName: \"kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.473632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.473583 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvv97\" (UniqueName: \"kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.473632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.473592 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdw8\" (UniqueName: \"kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.474939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.474918 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjqk\" (UniqueName: \"kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.475115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.475103 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:00.488039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.488019 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:00.508940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.508839 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:00.527761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.527405 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:00.560208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.560172 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.561013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.560990 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcz7\" (UniqueName: \"kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7\") pod \"maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.576021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.574747 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.583483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.583222 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:00.585652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.585626 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcz7\" (UniqueName: \"kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7\") pod \"maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:00.590738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.590232 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:00.623778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.623474 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:00.625890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.625812 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:00.634813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.634778 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:00.707268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:00.706544 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:01.014472 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.014440 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e05508d_5543_4f63_8303_369e5b276e93.slice/crio-3ca41255074098d288a2fd64f15c21d5b447dfa865064bbae7b850ae001520a8 WatchSource:0}: Error finding container 3ca41255074098d288a2fd64f15c21d5b447dfa865064bbae7b850ae001520a8: Status 404 returned error can't find the container with id 3ca41255074098d288a2fd64f15c21d5b447dfa865064bbae7b850ae001520a8 Oct 29 22:45:01.014938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.014883 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc"] Oct 29 22:45:01.038670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.038635 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487"] Oct 29 22:45:01.064289 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.064254 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757a1250_1588_47cd_9de6_46b0a0958a6e.slice/crio-019224b77c72da1968f383702ca4d39c2983bdfe7a5ea2739a417382fbbcc668 WatchSource:0}: Error finding container 019224b77c72da1968f383702ca4d39c2983bdfe7a5ea2739a417382fbbcc668: Status 404 returned error can't find the container with id 019224b77c72da1968f383702ca4d39c2983bdfe7a5ea2739a417382fbbcc668 Oct 29 22:45:01.104950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.104537 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd"] Oct 29 22:45:01.124311 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.124285 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a7e78b_aeae_4315_9c97_2d01ce3ff4d6.slice/crio-3c274caca05f1b8cdf6d61e3aac232f1638b368c1fa321909eeae038d23c4f9a WatchSource:0}: Error finding container 3c274caca05f1b8cdf6d61e3aac232f1638b368c1fa321909eeae038d23c4f9a: Status 404 returned error can't find the container with id 3c274caca05f1b8cdf6d61e3aac232f1638b368c1fa321909eeae038d23c4f9a Oct 29 22:45:01.145812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.145781 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6"] Oct 29 22:45:01.149762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.149733 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5"] Oct 29 22:45:01.156942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.156914 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d"] Oct 29 22:45:01.178614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.178586 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x"] Oct 29 22:45:01.274318 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.272989 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71d1651_adbf_4921_80f8_a616e5cb4b7d.slice/crio-a04cc1b740cf993979626b31c17bc9762f8597d6d4632ee1a256f82a4b58e14c WatchSource:0}: Error finding container a04cc1b740cf993979626b31c17bc9762f8597d6d4632ee1a256f82a4b58e14c: Status 404 returned error can't find the container with id a04cc1b740cf993979626b31c17bc9762f8597d6d4632ee1a256f82a4b58e14c Oct 29 22:45:01.309242 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.309213 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ba218c_89fd_4a1f_98a0_452f5e325e8b.slice/crio-e72dd9192244a72de5ebd147362e0630635011052c60c417291c7cb5a09ac7df WatchSource:0}: Error finding container e72dd9192244a72de5ebd147362e0630635011052c60c417291c7cb5a09ac7df: Status 404 returned error can't find the container with id e72dd9192244a72de5ebd147362e0630635011052c60c417291c7cb5a09ac7df Oct 29 22:45:01.310419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.310391 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h"] Oct 29 22:45:01.318452 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.318422 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35099194_653c_4dff_a88a_3920da55affb.slice/crio-81e0c0d6863424a1cfce8122d8e14d441b915a4ee2c1f877dd0b490f53b3a389 WatchSource:0}: Error finding container 81e0c0d6863424a1cfce8122d8e14d441b915a4ee2c1f877dd0b490f53b3a389: Status 404 returned error can't find the container with id 81e0c0d6863424a1cfce8122d8e14d441b915a4ee2c1f877dd0b490f53b3a389 Oct 29 22:45:01.319738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.319711 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp"] Oct 29 22:45:01.351166 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:01.351139 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7146a9_1a97_4446_b808_1d218616cf30.slice/crio-34060b26fe4557f9fa20dcb96549e825600dea53541a69185de9ab5e1ee66902 WatchSource:0}: Error finding container 34060b26fe4557f9fa20dcb96549e825600dea53541a69185de9ab5e1ee66902: Status 404 returned error can't find the container with id 34060b26fe4557f9fa20dcb96549e825600dea53541a69185de9ab5e1ee66902 Oct 29 22:45:01.352653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:01.352626 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq"] Oct 29 22:45:02.019633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.019599 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerStarted","Data":"79d5720dc628ff90cdc13fd9f4a0901bd23e0f17261e7b97d315297ca7b7ce92"} Oct 29 22:45:02.019879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.019666 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerStarted","Data":"81e0c0d6863424a1cfce8122d8e14d441b915a4ee2c1f877dd0b490f53b3a389"} Oct 29 22:45:02.021250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.021227 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerStarted","Data":"b770d2f9651a0d297bb532d8ebc3fbe99a153f248e988346e8c3f2435bd00b5d"} Oct 29 22:45:02.021357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.021270 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerStarted","Data":"3ca41255074098d288a2fd64f15c21d5b447dfa865064bbae7b850ae001520a8"} Oct 29 22:45:02.022749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.022679 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerStarted","Data":"0f8053f695067cb54530b3a983ab898af2d24861619774a10f759cf83427fade"} Oct 29 22:45:02.024269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.024235 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerStarted","Data":"34060b26fe4557f9fa20dcb96549e825600dea53541a69185de9ab5e1ee66902"} Oct 29 22:45:02.025606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.025584 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerStarted","Data":"b6209bebf2ee51978968426a94728e2afc541187dbc370f7357548ebef81644f"} Oct 29 22:45:02.027430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.027402 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerStarted","Data":"d4e0e7ce0f9c827956c7c6d682fbd572435fe3c053a2ab18f88cd0d4048b7a3f"} Oct 29 22:45:02.027518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.027465 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerStarted","Data":"f8d7b8f93320545343875b7f92729f3a5824c3af1b007791ed110dafb97551af"} Oct 29 22:45:02.029203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.029179 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerStarted","Data":"a399db072b074b868e433916b46b04f74b0de9ab371ec336d328b6b1749850d4"} Oct 29 22:45:02.029307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.029218 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerStarted","Data":"019224b77c72da1968f383702ca4d39c2983bdfe7a5ea2739a417382fbbcc668"} Oct 29 22:45:02.030861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.030838 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" event={"ID":"48ba218c-89fd-4a1f-98a0-452f5e325e8b","Type":"ContainerStarted","Data":"e72dd9192244a72de5ebd147362e0630635011052c60c417291c7cb5a09ac7df"} Oct 29 22:45:02.032634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.032615 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerStarted","Data":"0229d882bc2ad87d8206e6149d5af80651cba40fa643546ffd8e77c66d420566"} Oct 29 22:45:02.032745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.032658 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerStarted","Data":"a04cc1b740cf993979626b31c17bc9762f8597d6d4632ee1a256f82a4b58e14c"} Oct 29 22:45:02.034642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.034386 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerStarted","Data":"3c274caca05f1b8cdf6d61e3aac232f1638b368c1fa321909eeae038d23c4f9a"} Oct 29 22:45:02.073607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.073544 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" podStartSLOduration=2.073527966 podStartE2EDuration="2.073527966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.073070358 +0000 UTC m=+1768.967145280" watchObservedRunningTime="2026-10-29 22:45:02.073527966 +0000 UTC m=+1768.967602883" Oct 29 22:45:02.119491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.119341 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" podStartSLOduration=2.119323731 podStartE2EDuration="2.119323731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.118846115 +0000 UTC m=+1769.012921035" watchObservedRunningTime="2026-10-29 22:45:02.119323731 +0000 UTC m=+1769.013398653" Oct 29 22:45:02.166801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.166745 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" podStartSLOduration=2.166727995 podStartE2EDuration="2.166727995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.165447035 +0000 UTC m=+1769.059521954" watchObservedRunningTime="2026-10-29 22:45:02.166727995 +0000 UTC m=+1769.060802916" Oct 29 22:45:02.222943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.222883 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" podStartSLOduration=2.222864882 podStartE2EDuration="2.222864882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.220920867 +0000 UTC m=+1769.114995788" watchObservedRunningTime="2026-10-29 22:45:02.222864882 +0000 UTC m=+1769.116939802" Oct 29 22:45:02.287976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:02.287916 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" podStartSLOduration=2.287899454 podStartE2EDuration="2.287899454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.280358616 +0000 UTC m=+1769.174433552" watchObservedRunningTime="2026-10-29 22:45:02.287899454 +0000 UTC m=+1769.181974373" Oct 29 22:45:03.047308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.047270 5367 generic.go:356] "Generic (PLEG): container finished" podID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerID="0229d882bc2ad87d8206e6149d5af80651cba40fa643546ffd8e77c66d420566" exitCode=60 Oct 29 22:45:03.047468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.047369 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerDied","Data":"0229d882bc2ad87d8206e6149d5af80651cba40fa643546ffd8e77c66d420566"} Oct 29 22:45:03.047678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.047652 5367 scope.go:122] "RemoveContainer" containerID="0229d882bc2ad87d8206e6149d5af80651cba40fa643546ffd8e77c66d420566" Oct 29 22:45:03.049498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.049469 5367 generic.go:356] "Generic (PLEG): container finished" podID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerID="487770035a0b07b522ac475eec15aecef5ea1f6e9851c6ed02019f49221af142" exitCode=60 Oct 29 22:45:03.049749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.049637 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerDied","Data":"487770035a0b07b522ac475eec15aecef5ea1f6e9851c6ed02019f49221af142"} Oct 29 22:45:03.049826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.049811 5367 scope.go:122] "RemoveContainer" containerID="487770035a0b07b522ac475eec15aecef5ea1f6e9851c6ed02019f49221af142" Oct 29 22:45:03.051653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.051623 5367 generic.go:356] "Generic (PLEG): container finished" podID="35099194-653c-4dff-a88a-3920da55affb" containerID="79d5720dc628ff90cdc13fd9f4a0901bd23e0f17261e7b97d315297ca7b7ce92" exitCode=60 Oct 29 22:45:03.051789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.051653 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerDied","Data":"79d5720dc628ff90cdc13fd9f4a0901bd23e0f17261e7b97d315297ca7b7ce92"} Oct 29 22:45:03.051948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.051932 5367 scope.go:122] "RemoveContainer" containerID="79d5720dc628ff90cdc13fd9f4a0901bd23e0f17261e7b97d315297ca7b7ce92" Oct 29 22:45:03.053998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.053978 5367 generic.go:356] "Generic (PLEG): container finished" podID="1e05508d-5543-4f63-8303-369e5b276e93" containerID="b770d2f9651a0d297bb532d8ebc3fbe99a153f248e988346e8c3f2435bd00b5d" exitCode=60 Oct 29 22:45:03.054070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.054056 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerDied","Data":"b770d2f9651a0d297bb532d8ebc3fbe99a153f248e988346e8c3f2435bd00b5d"} Oct 29 22:45:03.054245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.054231 5367 scope.go:122] "RemoveContainer" containerID="b770d2f9651a0d297bb532d8ebc3fbe99a153f248e988346e8c3f2435bd00b5d" Oct 29 22:45:03.055734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.055677 5367 generic.go:356] "Generic (PLEG): container finished" podID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerID="0ebb4eb42b937dd66a2486230987b66ac742bddbf11978f534199aae8fbd636c" exitCode=60 Oct 29 22:45:03.055828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.055772 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerDied","Data":"0ebb4eb42b937dd66a2486230987b66ac742bddbf11978f534199aae8fbd636c"} Oct 29 22:45:03.056073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.056057 5367 scope.go:122] "RemoveContainer" containerID="0ebb4eb42b937dd66a2486230987b66ac742bddbf11978f534199aae8fbd636c" Oct 29 22:45:03.057332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.057312 5367 generic.go:356] "Generic (PLEG): container finished" podID="5f7146a9-1a97-4446-b808-1d218616cf30" containerID="e84f2608b32e1bfdcf9232799b8500da8bfabd1c962000f0f02112b949449ebe" exitCode=60 Oct 29 22:45:03.057417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.057387 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerDied","Data":"e84f2608b32e1bfdcf9232799b8500da8bfabd1c962000f0f02112b949449ebe"} Oct 29 22:45:03.057613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.057598 5367 scope.go:122] "RemoveContainer" containerID="e84f2608b32e1bfdcf9232799b8500da8bfabd1c962000f0f02112b949449ebe" Oct 29 22:45:03.059160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.059143 5367 generic.go:356] "Generic (PLEG): container finished" podID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerID="152ba5ee0fd1cca829544e0b478df13fab4aa522b2792abb768f754c8685d7c6" exitCode=60 Oct 29 22:45:03.059267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.059245 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerDied","Data":"152ba5ee0fd1cca829544e0b478df13fab4aa522b2792abb768f754c8685d7c6"} Oct 29 22:45:03.059482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.059461 5367 scope.go:122] "RemoveContainer" containerID="152ba5ee0fd1cca829544e0b478df13fab4aa522b2792abb768f754c8685d7c6" Oct 29 22:45:03.061675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.061655 5367 generic.go:356] "Generic (PLEG): container finished" podID="7115d2c4-0644-498c-9e4d-227f4d278139" containerID="d4e0e7ce0f9c827956c7c6d682fbd572435fe3c053a2ab18f88cd0d4048b7a3f" exitCode=60 Oct 29 22:45:03.061814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.061745 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerDied","Data":"d4e0e7ce0f9c827956c7c6d682fbd572435fe3c053a2ab18f88cd0d4048b7a3f"} Oct 29 22:45:03.062000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.061983 5367 scope.go:122] "RemoveContainer" containerID="d4e0e7ce0f9c827956c7c6d682fbd572435fe3c053a2ab18f88cd0d4048b7a3f" Oct 29 22:45:03.065686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.065668 5367 generic.go:356] "Generic (PLEG): container finished" podID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerID="a399db072b074b868e433916b46b04f74b0de9ab371ec336d328b6b1749850d4" exitCode=60 Oct 29 22:45:03.065814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.065753 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerDied","Data":"a399db072b074b868e433916b46b04f74b0de9ab371ec336d328b6b1749850d4"} Oct 29 22:45:03.066017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.066001 5367 scope.go:122] "RemoveContainer" containerID="a399db072b074b868e433916b46b04f74b0de9ab371ec336d328b6b1749850d4" Oct 29 22:45:03.067787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.067762 5367 generic.go:356] "Generic (PLEG): container finished" podID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerID="f5fea92815893a092585b41edd24d3e02c5dc49a4e0730c5225e199e8f9f5d48" exitCode=60 Oct 29 22:45:03.067877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.067795 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" event={"ID":"48ba218c-89fd-4a1f-98a0-452f5e325e8b","Type":"ContainerDied","Data":"f5fea92815893a092585b41edd24d3e02c5dc49a4e0730c5225e199e8f9f5d48"} Oct 29 22:45:03.068068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:03.068052 5367 scope.go:122] "RemoveContainer" containerID="f5fea92815893a092585b41edd24d3e02c5dc49a4e0730c5225e199e8f9f5d48" Oct 29 22:45:04.078749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.078724 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" event={"ID":"48ba218c-89fd-4a1f-98a0-452f5e325e8b","Type":"ContainerStarted","Data":"fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c"} Oct 29 22:45:04.080818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.080795 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerStarted","Data":"6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f"} Oct 29 22:45:04.083301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.083274 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerStarted","Data":"dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5"} Oct 29 22:45:04.085251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.085224 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerStarted","Data":"51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311"} Oct 29 22:45:04.087064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.087038 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerStarted","Data":"ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c"} Oct 29 22:45:04.088957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.088935 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerStarted","Data":"d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af"} Oct 29 22:45:04.091009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.090988 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerStarted","Data":"5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6"} Oct 29 22:45:04.093298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.093272 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerStarted","Data":"61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe"} Oct 29 22:45:04.095474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.095447 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerStarted","Data":"6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf"} Oct 29 22:45:04.098156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.098132 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerStarted","Data":"2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7"} Oct 29 22:45:04.134504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.134448 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" podStartSLOduration=4.134434652 podStartE2EDuration="4.134434652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.131864122 +0000 UTC m=+1771.025939044" watchObservedRunningTime="2026-10-29 22:45:04.134434652 +0000 UTC m=+1771.028509666" Oct 29 22:45:04.221133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.221023 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" podStartSLOduration=4.221009386 podStartE2EDuration="4.221009386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.219276948 +0000 UTC m=+1771.113351872" watchObservedRunningTime="2026-10-29 22:45:04.221009386 +0000 UTC m=+1771.115084304" Oct 29 22:45:04.277496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.277436 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" podStartSLOduration=4.277415993 podStartE2EDuration="4.277415993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.261992611 +0000 UTC m=+1771.156067535" watchObservedRunningTime="2026-10-29 22:45:04.277415993 +0000 UTC m=+1771.171490915" Oct 29 22:45:04.330318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.330249 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" podStartSLOduration=4.330228263 podStartE2EDuration="4.330228263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.312268467 +0000 UTC m=+1771.206343388" watchObservedRunningTime="2026-10-29 22:45:04.330228263 +0000 UTC m=+1771.224303183" Oct 29 22:45:04.452579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:04.452528 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" podStartSLOduration=4.452514256 podStartE2EDuration="4.452514256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.447153372 +0000 UTC m=+1771.341228294" watchObservedRunningTime="2026-10-29 22:45:04.452514256 +0000 UTC m=+1771.346589199" Oct 29 22:45:05.109182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.109146 5367 generic.go:356] "Generic (PLEG): container finished" podID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerID="6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f" exitCode=60 Oct 29 22:45:05.109723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.109323 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerDied","Data":"6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f"} Oct 29 22:45:05.109723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.109458 5367 scope.go:122] "RemoveContainer" containerID="0229d882bc2ad87d8206e6149d5af80651cba40fa643546ffd8e77c66d420566" Oct 29 22:45:05.110385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.110361 5367 scope.go:122] "RemoveContainer" containerID="6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f" Oct 29 22:45:05.110923 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.110897 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x_odh-ai-gateway-infra(d71d1651-adbf-4921-80f8-a616e5cb4b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" Oct 29 22:45:05.112110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.112080 5367 generic.go:356] "Generic (PLEG): container finished" podID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerID="dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5" exitCode=60 Oct 29 22:45:05.112203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.112140 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerDied","Data":"dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5"} Oct 29 22:45:05.112501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.112475 5367 scope.go:122] "RemoveContainer" containerID="dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5" Oct 29 22:45:05.112871 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.112845 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd_odh-ai-gateway-infra(75a7e78b-aeae-4315-9c97-2d01ce3ff4d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" Oct 29 22:45:05.114756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.114738 5367 generic.go:356] "Generic (PLEG): container finished" podID="35099194-653c-4dff-a88a-3920da55affb" containerID="51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311" exitCode=60 Oct 29 22:45:05.114862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.114768 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerDied","Data":"51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311"} Oct 29 22:45:05.115298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.115277 5367 scope.go:122] "RemoveContainer" containerID="51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311" Oct 29 22:45:05.115643 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.115622 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp_odh-ai-gateway-infra(35099194-653c-4dff-a88a-3920da55affb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" podUID="35099194-653c-4dff-a88a-3920da55affb" Oct 29 22:45:05.116917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.116899 5367 generic.go:356] "Generic (PLEG): container finished" podID="1e05508d-5543-4f63-8303-369e5b276e93" containerID="ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c" exitCode=60 Oct 29 22:45:05.117002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.116982 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerDied","Data":"ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c"} Oct 29 22:45:05.117243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.117225 5367 scope.go:122] "RemoveContainer" containerID="ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c" Oct 29 22:45:05.117574 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.117552 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc_odh-ai-gateway-infra(1e05508d-5543-4f63-8303-369e5b276e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" podUID="1e05508d-5543-4f63-8303-369e5b276e93" Oct 29 22:45:05.118930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.118909 5367 generic.go:356] "Generic (PLEG): container finished" podID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerID="d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af" exitCode=60 Oct 29 22:45:05.119032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.119021 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerDied","Data":"d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af"} Oct 29 22:45:05.119251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.119229 5367 scope.go:122] "RemoveContainer" containerID="d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af" Oct 29 22:45:05.119572 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.119544 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5_odh-ai-gateway-infra(4d0ff5c0-1ffa-4e9e-8046-242763cda94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" Oct 29 22:45:05.121130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.121108 5367 generic.go:356] "Generic (PLEG): container finished" podID="5f7146a9-1a97-4446-b808-1d218616cf30" containerID="5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6" exitCode=60 Oct 29 22:45:05.121204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.121183 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerDied","Data":"5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6"} Oct 29 22:45:05.121405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.121385 5367 scope.go:122] "RemoveContainer" containerID="5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6" Oct 29 22:45:05.121673 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.121652 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq_odh-ai-gateway-infra(5f7146a9-1a97-4446-b808-1d218616cf30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" Oct 29 22:45:05.123190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.123170 5367 generic.go:356] "Generic (PLEG): container finished" podID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerID="61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe" exitCode=60 Oct 29 22:45:05.123274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.123238 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerDied","Data":"61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe"} Oct 29 22:45:05.123434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.123418 5367 scope.go:122] "RemoveContainer" containerID="61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe" Oct 29 22:45:05.123682 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.123654 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6_odh-ai-gateway-infra(a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" Oct 29 22:45:05.125091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.125066 5367 generic.go:356] "Generic (PLEG): container finished" podID="7115d2c4-0644-498c-9e4d-227f4d278139" containerID="6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf" exitCode=60 Oct 29 22:45:05.125179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.125146 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerDied","Data":"6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf"} Oct 29 22:45:05.125359 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.125343 5367 scope.go:122] "RemoveContainer" containerID="6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf" Oct 29 22:45:05.125605 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.125582 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d_odh-ai-gateway-infra(7115d2c4-0644-498c-9e4d-227f4d278139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" Oct 29 22:45:05.126597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.126581 5367 scope.go:122] "RemoveContainer" containerID="487770035a0b07b522ac475eec15aecef5ea1f6e9851c6ed02019f49221af142" Oct 29 22:45:05.127014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.126997 5367 generic.go:356] "Generic (PLEG): container finished" podID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerID="2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7" exitCode=60 Oct 29 22:45:05.127088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.127070 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerDied","Data":"2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7"} Oct 29 22:45:05.127294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.127279 5367 scope.go:122] "RemoveContainer" containerID="2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7" Oct 29 22:45:05.127525 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.127508 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487_odh-ai-gateway-infra(757a1250-1588-47cd-9de6-46b0a0958a6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" Oct 29 22:45:05.129274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.129257 5367 generic.go:356] "Generic (PLEG): container finished" podID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerID="fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c" exitCode=60 Oct 29 22:45:05.129355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.129280 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" event={"ID":"48ba218c-89fd-4a1f-98a0-452f5e325e8b","Type":"ContainerDied","Data":"fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c"} Oct 29 22:45:05.129506 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.129492 5367 scope.go:122] "RemoveContainer" containerID="fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c" Oct 29 22:45:05.129728 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:05.129712 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h_odh-ai-gateway-infra(48ba218c-89fd-4a1f-98a0-452f5e325e8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" Oct 29 22:45:05.140986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.140968 5367 scope.go:122] "RemoveContainer" containerID="79d5720dc628ff90cdc13fd9f4a0901bd23e0f17261e7b97d315297ca7b7ce92" Oct 29 22:45:05.156396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.156375 5367 scope.go:122] "RemoveContainer" containerID="b770d2f9651a0d297bb532d8ebc3fbe99a153f248e988346e8c3f2435bd00b5d" Oct 29 22:45:05.171094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.171073 5367 scope.go:122] "RemoveContainer" containerID="0ebb4eb42b937dd66a2486230987b66ac742bddbf11978f534199aae8fbd636c" Oct 29 22:45:05.184851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.184829 5367 scope.go:122] "RemoveContainer" containerID="e84f2608b32e1bfdcf9232799b8500da8bfabd1c962000f0f02112b949449ebe" Oct 29 22:45:05.203391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.203333 5367 scope.go:122] "RemoveContainer" containerID="152ba5ee0fd1cca829544e0b478df13fab4aa522b2792abb768f754c8685d7c6" Oct 29 22:45:05.221163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.221145 5367 scope.go:122] "RemoveContainer" containerID="d4e0e7ce0f9c827956c7c6d682fbd572435fe3c053a2ab18f88cd0d4048b7a3f" Oct 29 22:45:05.236345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.236325 5367 scope.go:122] "RemoveContainer" containerID="a399db072b074b868e433916b46b04f74b0de9ab371ec336d328b6b1749850d4" Oct 29 22:45:05.250540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:05.250519 5367 scope.go:122] "RemoveContainer" containerID="f5fea92815893a092585b41edd24d3e02c5dc49a4e0730c5225e199e8f9f5d48" Oct 29 22:45:06.140572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.140540 5367 scope.go:122] "RemoveContainer" containerID="51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311" Oct 29 22:45:06.141058 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.140902 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp_odh-ai-gateway-infra(35099194-653c-4dff-a88a-3920da55affb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" podUID="35099194-653c-4dff-a88a-3920da55affb" Oct 29 22:45:06.142285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.142263 5367 scope.go:122] "RemoveContainer" containerID="ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c" Oct 29 22:45:06.142601 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.142580 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc_odh-ai-gateway-infra(1e05508d-5543-4f63-8303-369e5b276e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" podUID="1e05508d-5543-4f63-8303-369e5b276e93" Oct 29 22:45:06.144074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.144054 5367 scope.go:122] "RemoveContainer" containerID="d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af" Oct 29 22:45:06.144339 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.144321 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5_odh-ai-gateway-infra(4d0ff5c0-1ffa-4e9e-8046-242763cda94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" Oct 29 22:45:06.145638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.145610 5367 scope.go:122] "RemoveContainer" containerID="5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6" Oct 29 22:45:06.145929 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.145908 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq_odh-ai-gateway-infra(5f7146a9-1a97-4446-b808-1d218616cf30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" Oct 29 22:45:06.147290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.147273 5367 scope.go:122] "RemoveContainer" containerID="61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe" Oct 29 22:45:06.147577 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.147556 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6_odh-ai-gateway-infra(a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" Oct 29 22:45:06.149015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.148995 5367 scope.go:122] "RemoveContainer" containerID="6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf" Oct 29 22:45:06.149255 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.149236 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d_odh-ai-gateway-infra(7115d2c4-0644-498c-9e4d-227f4d278139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" Oct 29 22:45:06.150621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.150604 5367 scope.go:122] "RemoveContainer" containerID="2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7" Oct 29 22:45:06.150899 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.150876 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487_odh-ai-gateway-infra(757a1250-1588-47cd-9de6-46b0a0958a6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" Oct 29 22:45:06.152286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.152269 5367 scope.go:122] "RemoveContainer" containerID="fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c" Oct 29 22:45:06.152559 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.152538 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h_odh-ai-gateway-infra(48ba218c-89fd-4a1f-98a0-452f5e325e8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" Oct 29 22:45:06.153989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.153972 5367 scope.go:122] "RemoveContainer" containerID="6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f" Oct 29 22:45:06.154249 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.154228 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x_odh-ai-gateway-infra(d71d1651-adbf-4921-80f8-a616e5cb4b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" Oct 29 22:45:06.155561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:06.155543 5367 scope.go:122] "RemoveContainer" containerID="dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5" Oct 29 22:45:06.155837 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:06.155819 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd_odh-ai-gateway-infra(75a7e78b-aeae-4315-9c97-2d01ce3ff4d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" Oct 29 22:45:11.426064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.425997 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh"] Oct 29 22:45:11.433642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.433601 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.439349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.439318 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh"] Oct 29 22:45:11.449912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.449885 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:11.531430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.531392 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.531622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.531492 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.531740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.531720 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9n7p\" (UniqueName: \"kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.633054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.633021 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9n7p\" (UniqueName: \"kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.633237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.633219 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.633282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.633240 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.634008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.633989 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.635751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.635732 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.652820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.652794 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9n7p\" (UniqueName: \"kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p\") pod \"maas-api-revoke-sub-5a650f20-cqtvh\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:11.755042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:11.754966 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:12.059672 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:12.059639 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253608fc_b2c2_4258_ad1f_3eb17f558fd8.slice/crio-e833c100328a0cf444e1ffa7611ed00ebdd0730744713158094d42315414d948 WatchSource:0}: Error finding container e833c100328a0cf444e1ffa7611ed00ebdd0730744713158094d42315414d948: Status 404 returned error can't find the container with id e833c100328a0cf444e1ffa7611ed00ebdd0730744713158094d42315414d948 Oct 29 22:45:12.060577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:12.060552 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh"] Oct 29 22:45:12.304235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:12.304194 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" event={"ID":"253608fc-b2c2-4258-ad1f-3eb17f558fd8","Type":"ContainerStarted","Data":"e833c100328a0cf444e1ffa7611ed00ebdd0730744713158094d42315414d948"} Oct 29 22:45:13.315694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:13.315660 5367 generic.go:356] "Generic (PLEG): container finished" podID="253608fc-b2c2-4258-ad1f-3eb17f558fd8" containerID="c3f179365f750d532f675e48ef943485bfea01b6e4fb7b159f88f20d8d1d970c" exitCode=0 Oct 29 22:45:13.316140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:13.315762 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" event={"ID":"253608fc-b2c2-4258-ad1f-3eb17f558fd8","Type":"ContainerDied","Data":"c3f179365f750d532f675e48ef943485bfea01b6e4fb7b159f88f20d8d1d970c"} Oct 29 22:45:14.508307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.508283 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:14.593865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.593774 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p\" (UniqueName: \"kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p\") pod \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " Oct 29 22:45:14.594033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.593912 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca\") pod \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " Oct 29 22:45:14.594161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.594141 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca\") pod \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\" (UID: \"253608fc-b2c2-4258-ad1f-3eb17f558fd8\") " Oct 29 22:45:14.594334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.594307 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca" pod "253608fc-b2c2-4258-ad1f-3eb17f558fd8" (UID: "253608fc-b2c2-4258-ad1f-3eb17f558fd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:14.595127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.595100 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:14.596092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.596068 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p" pod "253608fc-b2c2-4258-ad1f-3eb17f558fd8" (UID: "253608fc-b2c2-4258-ad1f-3eb17f558fd8"). InnerVolumeSpecName "kube-api-access-d9n7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:14.596310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.596287 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca" pod "253608fc-b2c2-4258-ad1f-3eb17f558fd8" (UID: "253608fc-b2c2-4258-ad1f-3eb17f558fd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:14.697230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.697200 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253608fc-b2c2-4258-ad1f-3eb17f558fd8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:14.697230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:14.697228 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9n7p\" (UniqueName: \"kubernetes.io/projected/253608fc-b2c2-4258-ad1f-3eb17f558fd8-kube-api-access-d9n7p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.214443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:15.214408 5367 scope.go:122] "RemoveContainer" containerID="6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f" Oct 29 22:45:15.335920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:15.335894 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" Oct 29 22:45:15.335920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:15.335901 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh" event={"ID":"253608fc-b2c2-4258-ad1f-3eb17f558fd8","Type":"ContainerDied","Data":"e833c100328a0cf444e1ffa7611ed00ebdd0730744713158094d42315414d948"} Oct 29 22:45:15.336126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:15.335957 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e833c100328a0cf444e1ffa7611ed00ebdd0730744713158094d42315414d948" Oct 29 22:45:16.215119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:16.215089 5367 scope.go:122] "RemoveContainer" containerID="51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311" Oct 29 22:45:16.215589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:16.215367 5367 scope.go:122] "RemoveContainer" containerID="d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af" Oct 29 22:45:16.215589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:16.215466 5367 scope.go:122] "RemoveContainer" containerID="5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6" Oct 29 22:45:16.215589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:16.215584 5367 scope.go:122] "RemoveContainer" containerID="fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c" Oct 29 22:45:16.346774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:16.346748 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerStarted","Data":"8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a"} Oct 29 22:45:16.504824 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:16.504790 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71d1651_adbf_4921_80f8_a616e5cb4b7d.slice/crio-conmon-8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:45:16.504824 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:16.504822 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71d1651_adbf_4921_80f8_a616e5cb4b7d.slice/crio-conmon-8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:45:17.215453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.215423 5367 scope.go:122] "RemoveContainer" containerID="6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf" Oct 29 22:45:17.215956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.215627 5367 scope.go:122] "RemoveContainer" containerID="61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe" Oct 29 22:45:17.360211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.360179 5367 generic.go:356] "Generic (PLEG): container finished" podID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerID="ac752dbd66d7af554848b2fa9ab1310d7706e2909668d5fbb54ab2d1855c8454" exitCode=60 Oct 29 22:45:17.360338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.360254 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" event={"ID":"48ba218c-89fd-4a1f-98a0-452f5e325e8b","Type":"ContainerDied","Data":"ac752dbd66d7af554848b2fa9ab1310d7706e2909668d5fbb54ab2d1855c8454"} Oct 29 22:45:17.360338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.360327 5367 scope.go:122] "RemoveContainer" containerID="fb8638a212be00ed9331712ee6403445b6234b4a086e2d5f21a5a68c78258a9c" Oct 29 22:45:17.360841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.360780 5367 scope.go:122] "RemoveContainer" containerID="ac752dbd66d7af554848b2fa9ab1310d7706e2909668d5fbb54ab2d1855c8454" Oct 29 22:45:17.361267 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:17.361242 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h_odh-ai-gateway-infra(48ba218c-89fd-4a1f-98a0-452f5e325e8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" Oct 29 22:45:17.363941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.363920 5367 generic.go:356] "Generic (PLEG): container finished" podID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerID="8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a" exitCode=60 Oct 29 22:45:17.364038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.364013 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerDied","Data":"8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a"} Oct 29 22:45:17.364432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.364412 5367 scope.go:122] "RemoveContainer" containerID="8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a" Oct 29 22:45:17.364973 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:17.364944 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x_odh-ai-gateway-infra(d71d1651-adbf-4921-80f8-a616e5cb4b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" Oct 29 22:45:17.366477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.366452 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerStarted","Data":"b43aff74ebfe0f949f202f936f5453094d657739fefca5d33f75ce63ebfd698b"} Oct 29 22:45:17.366930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.366904 5367 scope.go:122] "RemoveContainer" containerID="b43aff74ebfe0f949f202f936f5453094d657739fefca5d33f75ce63ebfd698b" Oct 29 22:45:17.367280 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:17.367257 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp_odh-ai-gateway-infra(35099194-653c-4dff-a88a-3920da55affb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" podUID="35099194-653c-4dff-a88a-3920da55affb" Oct 29 22:45:17.368809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.368784 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerStarted","Data":"ab71ffa3e1f4ce4e92e31890da3e81328840249a84047d2c980b8c0cf0932c15"} Oct 29 22:45:17.371000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.370977 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerStarted","Data":"08c1d743d41f59f354a140d6a42ad2140286f6f125a9c8281da4b87688a2b53b"} Oct 29 22:45:17.371349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.371332 5367 scope.go:122] "RemoveContainer" containerID="08c1d743d41f59f354a140d6a42ad2140286f6f125a9c8281da4b87688a2b53b" Oct 29 22:45:17.371614 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:17.371596 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq_odh-ai-gateway-infra(5f7146a9-1a97-4446-b808-1d218616cf30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" Oct 29 22:45:17.404535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.404512 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x"] Oct 29 22:45:17.560971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:17.560943 5367 scope.go:122] "RemoveContainer" containerID="6a140e1e4fbed225f2ca8336049bba79f705f1984209696ee591cbd2efdfdd6f" Oct 29 22:45:18.215774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.215062 5367 scope.go:122] "RemoveContainer" containerID="2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7" Oct 29 22:45:18.216327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.216239 5367 scope.go:122] "RemoveContainer" containerID="dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5" Oct 29 22:45:18.275208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.275179 5367 scope.go:122] "RemoveContainer" containerID="ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c" Oct 29 22:45:18.525187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.525156 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h"] Oct 29 22:45:18.545791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.533113 5367 generic.go:356] "Generic (PLEG): container finished" podID="35099194-653c-4dff-a88a-3920da55affb" containerID="b43aff74ebfe0f949f202f936f5453094d657739fefca5d33f75ce63ebfd698b" exitCode=60 Oct 29 22:45:18.545791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.533256 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerDied","Data":"b43aff74ebfe0f949f202f936f5453094d657739fefca5d33f75ce63ebfd698b"} Oct 29 22:45:18.545791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.533309 5367 scope.go:122] "RemoveContainer" containerID="51c22fae61206b3d48af391828ba71f98e793561ac25b65b8346654052436311" Oct 29 22:45:18.545791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.533813 5367 scope.go:122] "RemoveContainer" containerID="b43aff74ebfe0f949f202f936f5453094d657739fefca5d33f75ce63ebfd698b" Oct 29 22:45:18.545791 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:18.534236 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp_odh-ai-gateway-infra(35099194-653c-4dff-a88a-3920da55affb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" podUID="35099194-653c-4dff-a88a-3920da55affb" Oct 29 22:45:18.549315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.548425 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp"] Oct 29 22:45:18.553863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.553832 5367 generic.go:356] "Generic (PLEG): container finished" podID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerID="ab71ffa3e1f4ce4e92e31890da3e81328840249a84047d2c980b8c0cf0932c15" exitCode=60 Oct 29 22:45:18.553971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.553953 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerDied","Data":"ab71ffa3e1f4ce4e92e31890da3e81328840249a84047d2c980b8c0cf0932c15"} Oct 29 22:45:18.554467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.554443 5367 scope.go:122] "RemoveContainer" containerID="ab71ffa3e1f4ce4e92e31890da3e81328840249a84047d2c980b8c0cf0932c15" Oct 29 22:45:18.555878 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:18.555852 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5_odh-ai-gateway-infra(4d0ff5c0-1ffa-4e9e-8046-242763cda94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" Oct 29 22:45:18.559094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.558725 5367 generic.go:356] "Generic (PLEG): container finished" podID="5f7146a9-1a97-4446-b808-1d218616cf30" containerID="08c1d743d41f59f354a140d6a42ad2140286f6f125a9c8281da4b87688a2b53b" exitCode=60 Oct 29 22:45:18.559094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.558852 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerDied","Data":"08c1d743d41f59f354a140d6a42ad2140286f6f125a9c8281da4b87688a2b53b"} Oct 29 22:45:18.559805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.559739 5367 scope.go:122] "RemoveContainer" containerID="08c1d743d41f59f354a140d6a42ad2140286f6f125a9c8281da4b87688a2b53b" Oct 29 22:45:18.561271 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:18.561238 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq_odh-ai-gateway-infra(5f7146a9-1a97-4446-b808-1d218616cf30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" Oct 29 22:45:18.567626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.566474 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq"] Oct 29 22:45:18.573136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.572840 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerStarted","Data":"59de3566cda37f42151ad102abd4c56addabb38ffac73d6b8b14d5f27f5dd182"} Oct 29 22:45:18.580770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.580745 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerStarted","Data":"f4d2935c85dbb120757866ab56fb13e21d5c66564a5cc9ad5f35a1bb2ab13770"} Oct 29 22:45:18.581237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.581213 5367 scope.go:122] "RemoveContainer" containerID="f4d2935c85dbb120757866ab56fb13e21d5c66564a5cc9ad5f35a1bb2ab13770" Oct 29 22:45:18.581709 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:18.581668 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d_odh-ai-gateway-infra(7115d2c4-0644-498c-9e4d-227f4d278139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" Oct 29 22:45:18.600556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.600476 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5"] Oct 29 22:45:18.638449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.638388 5367 scope.go:122] "RemoveContainer" containerID="d0e9afb0011d27e7cbc9a568bd5e1020c159622515b4f99f5cbbc8c08bb8d5af" Oct 29 22:45:18.663832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.663808 5367 scope.go:122] "RemoveContainer" containerID="5c719243ac12fc6a236248eca7212284057ae7e8bc8ba572d927055b3bd9c5b6" Oct 29 22:45:18.894662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.894637 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:18.898462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.898441 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:18.947075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.947046 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca\") pod \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " Oct 29 22:45:18.947246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.947218 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca\") pod \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " Oct 29 22:45:18.947386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.947372 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr\" (UniqueName: \"kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr\") pod \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\" (UID: \"d71d1651-adbf-4921-80f8-a616e5cb4b7d\") " Oct 29 22:45:18.947489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.947406 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca" pod "48ba218c-89fd-4a1f-98a0-452f5e325e8b" (UID: "48ba218c-89fd-4a1f-98a0-452f5e325e8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.947555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.947502 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7\" (UniqueName: \"kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7\") pod \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\" (UID: \"48ba218c-89fd-4a1f-98a0-452f5e325e8b\") " Oct 29 22:45:18.947607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.947547 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca" pod "d71d1651-adbf-4921-80f8-a616e5cb4b7d" (UID: "d71d1651-adbf-4921-80f8-a616e5cb4b7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.948516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.948491 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba218c-89fd-4a1f-98a0-452f5e325e8b-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.948632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.948518 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71d1651-adbf-4921-80f8-a616e5cb4b7d-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.950175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.950153 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr" pod "d71d1651-adbf-4921-80f8-a616e5cb4b7d" (UID: "d71d1651-adbf-4921-80f8-a616e5cb4b7d"). InnerVolumeSpecName "kube-api-access-jngzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.950278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:18.950191 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7" pod "48ba218c-89fd-4a1f-98a0-452f5e325e8b" (UID: "48ba218c-89fd-4a1f-98a0-452f5e325e8b"). InnerVolumeSpecName "kube-api-access-jvcz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.064874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.064838 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngzr\" (UniqueName: \"kubernetes.io/projected/d71d1651-adbf-4921-80f8-a616e5cb4b7d-kube-api-access-jngzr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.064874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.064875 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcz7\" (UniqueName: \"kubernetes.io/projected/48ba218c-89fd-4a1f-98a0-452f5e325e8b-kube-api-access-jvcz7\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.599599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.599564 5367 generic.go:356] "Generic (PLEG): container finished" podID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerID="59de3566cda37f42151ad102abd4c56addabb38ffac73d6b8b14d5f27f5dd182" exitCode=60 Oct 29 22:45:19.600004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.599646 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerDied","Data":"59de3566cda37f42151ad102abd4c56addabb38ffac73d6b8b14d5f27f5dd182"} Oct 29 22:45:19.600004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.599738 5367 scope.go:122] "RemoveContainer" containerID="61ba964be9eb68c8a5bf4153d28a870930066d121a6ba1274dd2bc201f051cfe" Oct 29 22:45:19.600277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.600258 5367 scope.go:122] "RemoveContainer" containerID="59de3566cda37f42151ad102abd4c56addabb38ffac73d6b8b14d5f27f5dd182" Oct 29 22:45:19.600782 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:19.600755 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6_odh-ai-gateway-infra(a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" Oct 29 22:45:19.602184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.602165 5367 generic.go:356] "Generic (PLEG): container finished" podID="7115d2c4-0644-498c-9e4d-227f4d278139" containerID="f4d2935c85dbb120757866ab56fb13e21d5c66564a5cc9ad5f35a1bb2ab13770" exitCode=60 Oct 29 22:45:19.602278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.602221 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerDied","Data":"f4d2935c85dbb120757866ab56fb13e21d5c66564a5cc9ad5f35a1bb2ab13770"} Oct 29 22:45:19.602753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.602734 5367 scope.go:122] "RemoveContainer" containerID="f4d2935c85dbb120757866ab56fb13e21d5c66564a5cc9ad5f35a1bb2ab13770" Oct 29 22:45:19.603127 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:19.603108 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d_odh-ai-gateway-infra(7115d2c4-0644-498c-9e4d-227f4d278139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" Oct 29 22:45:19.604509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.604490 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerStarted","Data":"9877c927160d14dd080d01ccdbd5371b8b22b25864b83f51c671056a68d4a860"} Oct 29 22:45:19.605982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.605966 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" Oct 29 22:45:19.606094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.606017 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h" event={"ID":"48ba218c-89fd-4a1f-98a0-452f5e325e8b","Type":"ContainerDied","Data":"e72dd9192244a72de5ebd147362e0630635011052c60c417291c7cb5a09ac7df"} Oct 29 22:45:19.607827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.607800 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" event={"ID":"d71d1651-adbf-4921-80f8-a616e5cb4b7d","Type":"ContainerDied","Data":"a04cc1b740cf993979626b31c17bc9762f8597d6d4632ee1a256f82a4b58e14c"} Oct 29 22:45:19.607929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.607904 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x" Oct 29 22:45:19.610017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.609996 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerStarted","Data":"16eb4c7893a10a2cf8a8ba5cb6eb3b5006972c6107eb5f3ab90624aebf975902"} Oct 29 22:45:19.614813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.614791 5367 generic.go:356] "Generic (PLEG): container finished" podID="1e05508d-5543-4f63-8303-369e5b276e93" containerID="d4c493e043fe73bdacfae5cdd6cfc4c85263fd28712e44b5f734df488fba13b2" exitCode=60 Oct 29 22:45:19.614920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.614852 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerDied","Data":"d4c493e043fe73bdacfae5cdd6cfc4c85263fd28712e44b5f734df488fba13b2"} Oct 29 22:45:19.615246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.615228 5367 scope.go:122] "RemoveContainer" containerID="d4c493e043fe73bdacfae5cdd6cfc4c85263fd28712e44b5f734df488fba13b2" Oct 29 22:45:19.615567 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:19.615543 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc_odh-ai-gateway-infra(1e05508d-5543-4f63-8303-369e5b276e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" podUID="1e05508d-5543-4f63-8303-369e5b276e93" Oct 29 22:45:19.629822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.629799 5367 scope.go:122] "RemoveContainer" containerID="6a970fe48ee9b2b96d1a963cc44fbb6701279de17cf7b44ff6796f9c8b34badf" Oct 29 22:45:19.673076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.673058 5367 scope.go:122] "RemoveContainer" containerID="ac752dbd66d7af554848b2fa9ab1310d7706e2909668d5fbb54ab2d1855c8454" Oct 29 22:45:19.680139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.680116 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6"] Oct 29 22:45:19.755314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.755236 5367 scope.go:122] "RemoveContainer" containerID="8b6d4b1e45e849960501fec6de2a94deeac9f1b2a5b92a012b1e1071f2fbf78a" Oct 29 22:45:19.868921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.863927 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k"] Oct 29 22:45:19.868921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.864184 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" containerID="cri-o://aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc" gracePeriod=30 Oct 29 22:45:19.873289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.873268 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n"] Oct 29 22:45:19.873733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.873693 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" containerID="cri-o://07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf" gracePeriod=30 Oct 29 22:45:19.875760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.875693 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:19.875943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.875921 5367 scope.go:122] "RemoveContainer" containerID="ee45fc7e8b16c1a12ecf1543b01e5e84aae5ca76fec1a765afcd657fb14d3c4c" Oct 29 22:45:19.916679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.916584 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d"] Oct 29 22:45:19.925487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.924814 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl"] Oct 29 22:45:19.925487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.925045 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" containerID="cri-o://6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c" gracePeriod=30 Oct 29 22:45:19.934378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.933965 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:19.934378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.934048 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd"] Oct 29 22:45:19.934378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.934225 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" containerID="cri-o://2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0" gracePeriod=30 Oct 29 22:45:19.948031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.944014 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:19.952200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.952142 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x"] Oct 29 22:45:19.952379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.952351 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" containerID="cri-o://000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108" gracePeriod=30 Oct 29 22:45:19.959728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.959130 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86"] Oct 29 22:45:19.959728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.959336 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" containerID="cri-o://dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b" gracePeriod=30 Oct 29 22:45:19.986682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.986643 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk\" (UniqueName: \"kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk\") pod \"5f7146a9-1a97-4446-b808-1d218616cf30\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " Oct 29 22:45:19.987193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.987168 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca\") pod \"35099194-653c-4dff-a88a-3920da55affb\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " Oct 29 22:45:19.987401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.987359 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97\" (UniqueName: \"kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97\") pod \"35099194-653c-4dff-a88a-3920da55affb\" (UID: \"35099194-653c-4dff-a88a-3920da55affb\") " Oct 29 22:45:19.987691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.987673 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca\") pod \"5f7146a9-1a97-4446-b808-1d218616cf30\" (UID: \"5f7146a9-1a97-4446-b808-1d218616cf30\") " Oct 29 22:45:19.991413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:19.991388 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca" pod "5f7146a9-1a97-4446-b808-1d218616cf30" (UID: "5f7146a9-1a97-4446-b808-1d218616cf30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.002498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.002473 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk" pod "5f7146a9-1a97-4446-b808-1d218616cf30" (UID: "5f7146a9-1a97-4446-b808-1d218616cf30"). InnerVolumeSpecName "kube-api-access-lfjqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.013724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.011918 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca" pod "35099194-653c-4dff-a88a-3920da55affb" (UID: "35099194-653c-4dff-a88a-3920da55affb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.125391 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca\") pod \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.125460 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx\" (UniqueName: \"kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx\") pod \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\" (UID: \"4d0ff5c0-1ffa-4e9e-8046-242763cda94c\") " Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.126574 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjqk\" (UniqueName: \"kubernetes.io/projected/5f7146a9-1a97-4446-b808-1d218616cf30-kube-api-access-lfjqk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.126600 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35099194-653c-4dff-a88a-3920da55affb-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.126616 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7146a9-1a97-4446-b808-1d218616cf30-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.129502 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca" pod "4d0ff5c0-1ffa-4e9e-8046-242763cda94c" (UID: "4d0ff5c0-1ffa-4e9e-8046-242763cda94c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.136722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.129714 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97" pod "35099194-653c-4dff-a88a-3920da55affb" (UID: "35099194-653c-4dff-a88a-3920da55affb"). InnerVolumeSpecName "kube-api-access-cvv97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.151722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.146281 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx" pod "4d0ff5c0-1ffa-4e9e-8046-242763cda94c" (UID: "4d0ff5c0-1ffa-4e9e-8046-242763cda94c"). InnerVolumeSpecName "kube-api-access-slnzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.199618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.196747 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h"] Oct 29 22:45:20.206968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.206892 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-379a6d-29888565-6fh4h"] Oct 29 22:45:20.233492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.229917 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvv97\" (UniqueName: \"kubernetes.io/projected/35099194-653c-4dff-a88a-3920da55affb-kube-api-access-cvv97\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.235004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.234956 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.235004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.234984 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnzx\" (UniqueName: \"kubernetes.io/projected/4d0ff5c0-1ffa-4e9e-8046-242763cda94c-kube-api-access-slnzx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.324848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.324816 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x"] Oct 29 22:45:20.338815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.338788 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888565-ldv6x"] Oct 29 22:45:20.729346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.729320 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" Oct 29 22:45:20.730114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.729338 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp" event={"ID":"35099194-653c-4dff-a88a-3920da55affb","Type":"ContainerDied","Data":"81e0c0d6863424a1cfce8122d8e14d441b915a4ee2c1f877dd0b490f53b3a389"} Oct 29 22:45:20.730114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.730050 5367 scope.go:122] "RemoveContainer" containerID="b43aff74ebfe0f949f202f936f5453094d657739fefca5d33f75ce63ebfd698b" Oct 29 22:45:20.733691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.733665 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" event={"ID":"4d0ff5c0-1ffa-4e9e-8046-242763cda94c","Type":"ContainerDied","Data":"0f8053f695067cb54530b3a983ab898af2d24861619774a10f759cf83427fade"} Oct 29 22:45:20.733889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.733806 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5" Oct 29 22:45:20.745394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.745373 5367 scope.go:122] "RemoveContainer" containerID="ab71ffa3e1f4ce4e92e31890da3e81328840249a84047d2c980b8c0cf0932c15" Oct 29 22:45:20.746262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.746247 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" Oct 29 22:45:20.746816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.746786 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq" event={"ID":"5f7146a9-1a97-4446-b808-1d218616cf30","Type":"ContainerDied","Data":"34060b26fe4557f9fa20dcb96549e825600dea53541a69185de9ab5e1ee66902"} Oct 29 22:45:20.767307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.767276 5367 scope.go:122] "RemoveContainer" containerID="08c1d743d41f59f354a140d6a42ad2140286f6f125a9c8281da4b87688a2b53b" Oct 29 22:45:20.772624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.772593 5367 generic.go:356] "Generic (PLEG): container finished" podID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerID="9877c927160d14dd080d01ccdbd5371b8b22b25864b83f51c671056a68d4a860" exitCode=60 Oct 29 22:45:20.772800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.772777 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerDied","Data":"9877c927160d14dd080d01ccdbd5371b8b22b25864b83f51c671056a68d4a860"} Oct 29 22:45:20.773333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.773312 5367 scope.go:122] "RemoveContainer" containerID="9877c927160d14dd080d01ccdbd5371b8b22b25864b83f51c671056a68d4a860" Oct 29 22:45:20.775143 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:20.773899 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487_odh-ai-gateway-infra(757a1250-1588-47cd-9de6-46b0a0958a6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" Oct 29 22:45:20.785375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.785339 5367 generic.go:356] "Generic (PLEG): container finished" podID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerID="16eb4c7893a10a2cf8a8ba5cb6eb3b5006972c6107eb5f3ab90624aebf975902" exitCode=60 Oct 29 22:45:20.785514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.785403 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerDied","Data":"16eb4c7893a10a2cf8a8ba5cb6eb3b5006972c6107eb5f3ab90624aebf975902"} Oct 29 22:45:20.785834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.785814 5367 scope.go:122] "RemoveContainer" containerID="16eb4c7893a10a2cf8a8ba5cb6eb3b5006972c6107eb5f3ab90624aebf975902" Oct 29 22:45:20.786158 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:20.786137 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd_odh-ai-gateway-infra(75a7e78b-aeae-4315-9c97-2d01ce3ff4d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" Oct 29 22:45:20.788998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.788976 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc"] Oct 29 22:45:20.792520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.792503 5367 scope.go:122] "RemoveContainer" containerID="2632ded7dea332d2a9d40f3d7e71f3ef6aaca344ebd9782faf404be71b662ee7" Oct 29 22:45:20.812446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.811906 5367 scope.go:122] "RemoveContainer" containerID="dd278aeadb075928fcb3b40a9c9e4f883305e0a60db596b9e646f84571a091b5" Oct 29 22:45:20.886316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.886285 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487"] Oct 29 22:45:20.967404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.967377 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp"] Oct 29 22:45:20.976799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.976770 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e526ec-29888565-xqzdp"] Oct 29 22:45:20.995537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:20.995509 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:21.011603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.011549 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq"] Oct 29 22:45:21.020738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.020693 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888565-hfkfq"] Oct 29 22:45:21.083134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.083069 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca\") pod \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " Oct 29 22:45:21.083282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.083147 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8\" (UniqueName: \"kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8\") pod \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\" (UID: \"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41\") " Oct 29 22:45:21.083468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.083441 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca" pod "a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" (UID: "a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.083971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.083948 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.085216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.085198 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:21.086110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.086090 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8" pod "a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" (UID: "a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41"). InnerVolumeSpecName "kube-api-access-gkdw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.090907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.090887 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:21.185535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.185499 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca\") pod \"7115d2c4-0644-498c-9e4d-227f4d278139\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " Oct 29 22:45:21.186621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.186594 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf\" (UniqueName: \"kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf\") pod \"1e05508d-5543-4f63-8303-369e5b276e93\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " Oct 29 22:45:21.186621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.186038 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca" pod "7115d2c4-0644-498c-9e4d-227f4d278139" (UID: "7115d2c4-0644-498c-9e4d-227f4d278139"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.186888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.186651 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb\" (UniqueName: \"kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb\") pod \"7115d2c4-0644-498c-9e4d-227f4d278139\" (UID: \"7115d2c4-0644-498c-9e4d-227f4d278139\") " Oct 29 22:45:21.187048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.187015 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca\") pod \"1e05508d-5543-4f63-8303-369e5b276e93\" (UID: \"1e05508d-5543-4f63-8303-369e5b276e93\") " Oct 29 22:45:21.187641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.187607 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca" pod "1e05508d-5543-4f63-8303-369e5b276e93" (UID: "1e05508d-5543-4f63-8303-369e5b276e93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.187992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.187968 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d2c4-0644-498c-9e4d-227f4d278139-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.188083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.187998 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkdw8\" (UniqueName: \"kubernetes.io/projected/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41-kube-api-access-gkdw8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.188083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.188014 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e05508d-5543-4f63-8303-369e5b276e93-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.188981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.188960 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf" pod "1e05508d-5543-4f63-8303-369e5b276e93" (UID: "1e05508d-5543-4f63-8303-369e5b276e93"). InnerVolumeSpecName "kube-api-access-lmggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.189850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.189832 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb" pod "7115d2c4-0644-498c-9e4d-227f4d278139" (UID: "7115d2c4-0644-498c-9e4d-227f4d278139"). InnerVolumeSpecName "kube-api-access-vt5fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.238226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.238194 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35099194-653c-4dff-a88a-3920da55affb" path="/var/lib/kubelet/pods/35099194-653c-4dff-a88a-3920da55affb/volumes" Oct 29 22:45:21.241654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.241627 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" path="/var/lib/kubelet/pods/48ba218c-89fd-4a1f-98a0-452f5e325e8b/volumes" Oct 29 22:45:21.242740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.242304 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" path="/var/lib/kubelet/pods/5f7146a9-1a97-4446-b808-1d218616cf30/volumes" Oct 29 22:45:21.242873 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.242841 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" path="/var/lib/kubelet/pods/d71d1651-adbf-4921-80f8-a616e5cb4b7d/volumes" Oct 29 22:45:21.244215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.244195 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5"] Oct 29 22:45:21.244315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.244230 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888565-tzxt5"] Oct 29 22:45:21.296528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.296494 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5fb\" (UniqueName: \"kubernetes.io/projected/7115d2c4-0644-498c-9e4d-227f4d278139-kube-api-access-vt5fb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.296528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.296527 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmggf\" (UniqueName: \"kubernetes.io/projected/1e05508d-5543-4f63-8303-369e5b276e93-kube-api-access-lmggf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.309121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.309091 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd"] Oct 29 22:45:21.326717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.326679 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc"] Oct 29 22:45:21.327192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.327126 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" containerID="cri-o://6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993" gracePeriod=30 Oct 29 22:45:21.804665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.804636 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" Oct 29 22:45:21.805282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.804636 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc" event={"ID":"1e05508d-5543-4f63-8303-369e5b276e93","Type":"ContainerDied","Data":"3ca41255074098d288a2fd64f15c21d5b447dfa865064bbae7b850ae001520a8"} Oct 29 22:45:21.805282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.804812 5367 scope.go:122] "RemoveContainer" containerID="d4c493e043fe73bdacfae5cdd6cfc4c85263fd28712e44b5f734df488fba13b2" Oct 29 22:45:21.808936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.808918 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" Oct 29 22:45:21.808936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.808927 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6" event={"ID":"a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41","Type":"ContainerDied","Data":"b6209bebf2ee51978968426a94728e2afc541187dbc370f7357548ebef81644f"} Oct 29 22:45:21.810541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.810514 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" Oct 29 22:45:21.810661 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.810517 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d" event={"ID":"7115d2c4-0644-498c-9e4d-227f4d278139","Type":"ContainerDied","Data":"f8d7b8f93320545343875b7f92729f3a5824c3af1b007791ed110dafb97551af"} Oct 29 22:45:21.817871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.817843 5367 scope.go:122] "RemoveContainer" containerID="59de3566cda37f42151ad102abd4c56addabb38ffac73d6b8b14d5f27f5dd182" Oct 29 22:45:21.829809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.829791 5367 scope.go:122] "RemoveContainer" containerID="f4d2935c85dbb120757866ab56fb13e21d5c66564a5cc9ad5f35a1bb2ab13770" Oct 29 22:45:21.917877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.917841 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6"] Oct 29 22:45:21.925667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.925641 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888565-v5jg6"] Oct 29 22:45:21.973025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.972525 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc"] Oct 29 22:45:21.980854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:21.980827 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-139eb7-29888565-fcjcc"] Oct 29 22:45:22.051128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.051101 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:22.054510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.054489 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:22.056438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.056362 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d"] Oct 29 22:45:22.061437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.061417 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888565-sbm5d"] Oct 29 22:45:22.215494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.215465 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca\") pod \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " Oct 29 22:45:22.215683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.215558 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr\" (UniqueName: \"kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr\") pod \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\" (UID: \"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6\") " Oct 29 22:45:22.215683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.215605 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp\" (UniqueName: \"kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp\") pod \"757a1250-1588-47cd-9de6-46b0a0958a6e\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " Oct 29 22:45:22.215683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.215645 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca\") pod \"757a1250-1588-47cd-9de6-46b0a0958a6e\" (UID: \"757a1250-1588-47cd-9de6-46b0a0958a6e\") " Oct 29 22:45:22.215992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.215962 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca" pod "75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" (UID: "75a7e78b-aeae-4315-9c97-2d01ce3ff4d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:22.216316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.216291 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca" pod "757a1250-1588-47cd-9de6-46b0a0958a6e" (UID: "757a1250-1588-47cd-9de6-46b0a0958a6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:22.217861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.217837 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp" pod "757a1250-1588-47cd-9de6-46b0a0958a6e" (UID: "757a1250-1588-47cd-9de6-46b0a0958a6e"). InnerVolumeSpecName "kube-api-access-9cvhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:22.218099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.218077 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr" pod "75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" (UID: "75a7e78b-aeae-4315-9c97-2d01ce3ff4d6"). InnerVolumeSpecName "kube-api-access-76phr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:22.218099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.218082 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.218278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.218120 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvhp\" (UniqueName: \"kubernetes.io/projected/757a1250-1588-47cd-9de6-46b0a0958a6e-kube-api-access-9cvhp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.218278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.218132 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a1250-1588-47cd-9de6-46b0a0958a6e-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.319452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.319364 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76phr\" (UniqueName: \"kubernetes.io/projected/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6-kube-api-access-76phr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.333288 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:22.333254 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:45:22.825306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.825280 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" Oct 29 22:45:22.825805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.825307 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd" event={"ID":"75a7e78b-aeae-4315-9c97-2d01ce3ff4d6","Type":"ContainerDied","Data":"3c274caca05f1b8cdf6d61e3aac232f1638b368c1fa321909eeae038d23c4f9a"} Oct 29 22:45:22.825805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.825368 5367 scope.go:122] "RemoveContainer" containerID="16eb4c7893a10a2cf8a8ba5cb6eb3b5006972c6107eb5f3ab90624aebf975902" Oct 29 22:45:22.831909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.831880 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" event={"ID":"757a1250-1588-47cd-9de6-46b0a0958a6e","Type":"ContainerDied","Data":"019224b77c72da1968f383702ca4d39c2983bdfe7a5ea2739a417382fbbcc668"} Oct 29 22:45:22.831909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.831904 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487" Oct 29 22:45:22.840985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.840965 5367 scope.go:122] "RemoveContainer" containerID="9877c927160d14dd080d01ccdbd5371b8b22b25864b83f51c671056a68d4a860" Oct 29 22:45:22.882874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.882849 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd"] Oct 29 22:45:22.889978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.889955 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e224cd-29888565-72qpd"] Oct 29 22:45:22.934460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.934436 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487"] Oct 29 22:45:22.939114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:22.939096 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888565-wz487"] Oct 29 22:45:23.227813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:23.227730 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e05508d-5543-4f63-8303-369e5b276e93" path="/var/lib/kubelet/pods/1e05508d-5543-4f63-8303-369e5b276e93/volumes" Oct 29 22:45:23.228191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:23.228178 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" path="/var/lib/kubelet/pods/4d0ff5c0-1ffa-4e9e-8046-242763cda94c/volumes" Oct 29 22:45:23.228608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:23.228589 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" path="/var/lib/kubelet/pods/7115d2c4-0644-498c-9e4d-227f4d278139/volumes" Oct 29 22:45:23.229341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:23.229325 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" path="/var/lib/kubelet/pods/757a1250-1588-47cd-9de6-46b0a0958a6e/volumes" Oct 29 22:45:23.229744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:23.229731 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" path="/var/lib/kubelet/pods/75a7e78b-aeae-4315-9c97-2d01ce3ff4d6/volumes" Oct 29 22:45:23.230130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:23.230117 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" path="/var/lib/kubelet/pods/a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41/volumes" Oct 29 22:45:24.056938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.056897 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:24.057455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.056960 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:24.549911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.549822 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:24.550073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.549901 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:24.551434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.551401 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:24.551546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.551450 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:24.667421 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.667386 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt"] Oct 29 22:45:24.668208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668191 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668209 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668222 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668227 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668240 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668244 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668253 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668258 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.668267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668269 5367 cpu_manager.go:414] "containerMap: removing container" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668273 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668283 5367 cpu_manager.go:414] "containerMap: removing container" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668287 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668293 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668298 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668306 5367 cpu_manager.go:414] "containerMap: removing container" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668310 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668315 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668319 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668325 5367 cpu_manager.go:414] "containerMap: removing container" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668330 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668338 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668343 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668350 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668354 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668366 5367 cpu_manager.go:414] "containerMap: removing container" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668371 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668377 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668381 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668389 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668393 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668399 5367 cpu_manager.go:414] "containerMap: removing container" podUID="253608fc-b2c2-4258-ad1f-3eb17f558fd8" containerName="revoke-keys" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668404 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253608fc-b2c2-4258-ad1f-3eb17f558fd8" containerName="revoke-keys" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668412 5367 cpu_manager.go:414] "containerMap: removing container" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668415 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668422 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668426 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668433 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668437 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668444 5367 cpu_manager.go:414] "containerMap: removing container" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668448 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668457 5367 cpu_manager.go:414] "containerMap: removing container" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668461 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668472 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668476 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668484 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668488 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668516 5367 cpu_manager.go:414] "containerMap: removing container" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668520 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668530 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:24.668517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668534 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668540 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668544 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668549 5367 cpu_manager.go:414] "containerMap: removing container" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668553 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668670 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668677 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668685 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668690 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668707 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668712 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668717 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668721 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668727 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668734 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668738 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668744 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668748 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98b2bcd-e1c8-4ba2-ac2b-01ddd40e5b41" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668755 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668762 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668768 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668773 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a7e78b-aeae-4315-9c97-2d01ce3ff4d6" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668779 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668784 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668792 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668797 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668803 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="253608fc-b2c2-4258-ad1f-3eb17f558fd8" containerName="revoke-keys" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668809 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0ff5c0-1ffa-4e9e-8046-242763cda94c" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668816 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.669787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.668821 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="35099194-653c-4dff-a88a-3920da55affb" containerName="cleanup" Oct 29 22:45:24.672812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.672796 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.695929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.695904 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt"] Oct 29 22:45:24.702793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.702764 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:24.702939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.702884 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:45:24.859437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.859401 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.859616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.859561 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.859616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.859586 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5j8\" (UniqueName: \"kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.961879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.961843 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.962055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.961970 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.962055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.962000 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5j8\" (UniqueName: \"kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.962684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.962658 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.964562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.964542 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.983289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.983259 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5j8\" (UniqueName: \"kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8\") pod \"maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:24.987067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:24.987026 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:25.308883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.308549 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt"] Oct 29 22:45:25.308883 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:25.308831 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc3c6e4_5221_4bb1_97bf_ec0ed027abdb.slice/crio-7ec0ce19651e2aedaab334b98cdc97f89e8557afb1bb0f5141073a907a9b70f8 WatchSource:0}: Error finding container 7ec0ce19651e2aedaab334b98cdc97f89e8557afb1bb0f5141073a907a9b70f8: Status 404 returned error can't find the container with id 7ec0ce19651e2aedaab334b98cdc97f89e8557afb1bb0f5141073a907a9b70f8 Oct 29 22:45:25.424456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.424428 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:45:25.428578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.428557 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565337 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565388 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565419 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565444 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565474 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565551 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565624 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565656 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565726 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565905 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565943 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.565998 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566101 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566174 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket\") pod \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\" (UID: \"bc578fbe-af4d-42d1-8da6-169c2e314e1e\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566208 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566272 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566306 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566336 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token\") pod \"5f992356-17a9-4281-943f-a567571fd9ef\" (UID: \"5f992356-17a9-4281-943f-a567571fd9ef\") " Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566561 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.566987 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.567273 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.567513 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568048 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568302 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568345 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568609 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568890 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568912 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568924 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568935 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568947 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568959 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568972 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.571469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.568984 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.573368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.572743 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "kube-api-access-4bn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.573752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.573681 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.574423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.574392 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.576049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.576012 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.587745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.587686 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:45:25.590454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.590387 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:45:25.594479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.594460 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:45:25.595124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.595081 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.595124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.595106 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.595323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.595110 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "kube-api-access-gntxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.595323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.595130 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.595609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.595584 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token" pod "bc578fbe-af4d-42d1-8da6-169c2e314e1e" (UID: "bc578fbe-af4d-42d1-8da6-169c2e314e1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.597145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.597127 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy" pod "5f992356-17a9-4281-943f-a567571fd9ef" (UID: "5f992356-17a9-4281-943f-a567571fd9ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.598750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.598689 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:45:25.671273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671242 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671274 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f992356-17a9-4281-943f-a567571fd9ef-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671288 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671301 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f992356-17a9-4281-943f-a567571fd9ef-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671314 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671331 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f992356-17a9-4281-943f-a567571fd9ef-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671343 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671355 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671367 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntxr\" (UniqueName: \"kubernetes.io/projected/5f992356-17a9-4281-943f-a567571fd9ef-kube-api-access-gntxr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.671523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.671380 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bn8b\" (UniqueName: \"kubernetes.io/projected/bc578fbe-af4d-42d1-8da6-169c2e314e1e-kube-api-access-4bn8b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.772835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.772802 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.772835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.772839 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.773082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.772864 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.773082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.772933 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.773082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.772974 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.773082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773027 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.773082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773053 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773112 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773143 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773152 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773193 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773222 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.773334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773307 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.773687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773320 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.773687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773453 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.773687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773484 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.773687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773521 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.773687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773640 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.773687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773666 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773715 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773748 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773858 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773884 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773936 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773962 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.774015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.773980 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774019 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774063 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774094 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket\") pod \"583be168-6bc5-4983-a98c-9bca968b4b97\" (UID: \"583be168-6bc5-4983-a98c-9bca968b4b97\") " Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774129 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774161 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774255 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.774368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774357 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.774730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774473 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.774730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.774559 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.775430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.775156 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.777573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.777470 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778024 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778088 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778130 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778190 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778249 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778292 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778337 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token\") pod \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\" (UID: \"d97d9d81-60f1-47f2-98fb-997c8c4de79d\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778377 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket\") pod \"2667a74a-dee8-425f-87be-3d9a39bba51e\" (UID: \"2667a74a-dee8-425f-87be-3d9a39bba51e\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.778432 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert\") pod \"f51b751f-722b-4001-95cd-04ce87fd5356\" (UID: \"f51b751f-722b-4001-95cd-04ce87fd5356\") " Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779475 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779492 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779501 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779511 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779519 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583be168-6bc5-4983-a98c-9bca968b4b97-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779527 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779535 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779543 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779551 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2667a74a-dee8-425f-87be-3d9a39bba51e-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779558 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779565 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.779746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.779604 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.780955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.780139 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "kube-api-access-8b5j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.780955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.780347 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.781474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.781223 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.781660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.781626 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.782200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.782174 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.782436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.782406 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.782554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.782533 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.783310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783064 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.783310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783110 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.783310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783123 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.783310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783114 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.783310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783220 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.783808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783336 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.783808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783353 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "kube-api-access-g7zjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.783808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783445 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.783808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783489 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.783808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.783576 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.784619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.784590 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.785778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.785743 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c" pod "f51b751f-722b-4001-95cd-04ce87fd5356" (UID: "f51b751f-722b-4001-95cd-04ce87fd5356"). InnerVolumeSpecName "kube-api-access-sws7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.785986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.785966 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy" pod "d97d9d81-60f1-47f2-98fb-997c8c4de79d" (UID: "d97d9d81-60f1-47f2-98fb-997c8c4de79d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.787093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.787063 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.787093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.787073 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.787376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.787179 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy" pod "2667a74a-dee8-425f-87be-3d9a39bba51e" (UID: "2667a74a-dee8-425f-87be-3d9a39bba51e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.788478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.788415 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq" pod "583be168-6bc5-4983-a98c-9bca968b4b97" (UID: "583be168-6bc5-4983-a98c-9bca968b4b97"). InnerVolumeSpecName "kube-api-access-6fjzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.875972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.873947 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" event={"ID":"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb","Type":"ContainerStarted","Data":"7ec0ce19651e2aedaab334b98cdc97f89e8557afb1bb0f5141073a907a9b70f8"} Oct 29 22:45:25.876392 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.876363 5367 generic.go:356] "Generic (PLEG): container finished" podID="f51b751f-722b-4001-95cd-04ce87fd5356" containerID="2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0" exitCode=0 Oct 29 22:45:25.877586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.876750 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" event={"ID":"f51b751f-722b-4001-95cd-04ce87fd5356","Type":"ContainerDied","Data":"2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0"} Oct 29 22:45:25.877586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.876809 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" event={"ID":"f51b751f-722b-4001-95cd-04ce87fd5356","Type":"ContainerDied","Data":"d37d2f32f424d51c5ca52c9cec87dbf611f6e425539fb8efaa230b881490d304"} Oct 29 22:45:25.877586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.876849 5367 scope.go:122] "RemoveContainer" containerID="2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0" Oct 29 22:45:25.877964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.877946 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd" Oct 29 22:45:25.879103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.879073 5367 generic.go:356] "Generic (PLEG): container finished" podID="583be168-6bc5-4983-a98c-9bca968b4b97" containerID="dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b" exitCode=0 Oct 29 22:45:25.879542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.879188 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" Oct 29 22:45:25.879542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.879190 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" event={"ID":"583be168-6bc5-4983-a98c-9bca968b4b97","Type":"ContainerDied","Data":"dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b"} Oct 29 22:45:25.879542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.879312 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86" event={"ID":"583be168-6bc5-4983-a98c-9bca968b4b97","Type":"ContainerDied","Data":"bbce4aaf4f807da225a327ede2677035cedf2667ed0b172feb36e4594c5ef28e"} Oct 29 22:45:25.882005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.881983 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882011 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sws7c\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-kube-api-access-sws7c\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882028 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882042 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882055 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882067 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zjh\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-kube-api-access-g7zjh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882163 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882176 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882187 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51b751f-722b-4001-95cd-04ce87fd5356-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882197 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882208 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882219 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882230 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51b751f-722b-4001-95cd-04ce87fd5356-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882242 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583be168-6bc5-4983-a98c-9bca968b4b97-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882252 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882263 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882273 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583be168-6bc5-4983-a98c-9bca968b4b97-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882320 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b5j2\" (UniqueName: \"kubernetes.io/projected/d97d9d81-60f1-47f2-98fb-997c8c4de79d-kube-api-access-8b5j2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882334 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2667a74a-dee8-425f-87be-3d9a39bba51e-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882345 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882356 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51b751f-722b-4001-95cd-04ce87fd5356-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882358 5367 generic.go:356] "Generic (PLEG): container finished" podID="5f992356-17a9-4281-943f-a567571fd9ef" containerID="07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf" exitCode=0 Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882458 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882367 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fjzq\" (UniqueName: \"kubernetes.io/projected/583be168-6bc5-4983-a98c-9bca968b4b97-kube-api-access-6fjzq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882497 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882514 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51b751f-722b-4001-95cd-04ce87fd5356-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882528 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d9d81-60f1-47f2-98fb-997c8c4de79d-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882546 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" event={"ID":"5f992356-17a9-4281-943f-a567571fd9ef","Type":"ContainerDied","Data":"07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf"} Oct 29 22:45:25.882782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.882586 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" event={"ID":"5f992356-17a9-4281-943f-a567571fd9ef","Type":"ContainerDied","Data":"60b7547df58e5d8bad983e34e6b70bb894d06e6918f5dd19aca86fed7bf88c40"} Oct 29 22:45:25.886423 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.886393 5367 generic.go:356] "Generic (PLEG): container finished" podID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerID="aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc" exitCode=0 Oct 29 22:45:25.886611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.886457 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" Oct 29 22:45:25.886817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.886480 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" event={"ID":"bc578fbe-af4d-42d1-8da6-169c2e314e1e","Type":"ContainerDied","Data":"aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc"} Oct 29 22:45:25.886906 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.886852 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k" event={"ID":"bc578fbe-af4d-42d1-8da6-169c2e314e1e","Type":"ContainerDied","Data":"401443bbf6bd1a0333d4b8ce075727fc0f3d96928ff5e32a384a9f56cf6e9017"} Oct 29 22:45:25.890288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.890263 5367 generic.go:356] "Generic (PLEG): container finished" podID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerID="6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c" exitCode=0 Oct 29 22:45:25.890396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.890319 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" Oct 29 22:45:25.890396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.890382 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" event={"ID":"d97d9d81-60f1-47f2-98fb-997c8c4de79d","Type":"ContainerDied","Data":"6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c"} Oct 29 22:45:25.890515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.890428 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl" event={"ID":"d97d9d81-60f1-47f2-98fb-997c8c4de79d","Type":"ContainerDied","Data":"9b5eebd1eeb812e9a5b518336baa7b2a1185d585bf7087d0db23b4c6ba038ee2"} Oct 29 22:45:25.893345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.893322 5367 generic.go:356] "Generic (PLEG): container finished" podID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerID="000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108" exitCode=0 Oct 29 22:45:25.893457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.893374 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" Oct 29 22:45:25.893457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.893386 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" event={"ID":"2667a74a-dee8-425f-87be-3d9a39bba51e","Type":"ContainerDied","Data":"000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108"} Oct 29 22:45:25.893457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.893421 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x" event={"ID":"2667a74a-dee8-425f-87be-3d9a39bba51e","Type":"ContainerDied","Data":"6fcb9a6b7c7445aebbbdd9350fb9e6a9c666845696b88c9b85789211e9dabe9e"} Oct 29 22:45:25.893694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.893592 5367 scope.go:122] "RemoveContainer" containerID="2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0" Oct 29 22:45:25.894100 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:25.894077 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0\": container with ID starting with 2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0 not found: ID does not exist" containerID="2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0" Oct 29 22:45:25.894234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.894213 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0"} err="failed to get container status \"2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0\": rpc error: code = NotFound desc = could not find container \"2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0\": container with ID starting with 2aabd3d56d0c0c8894358a1fa53de2c4bc3ece7b38556db1f30d23a555a15bc0 not found: ID does not exist" Oct 29 22:45:25.894344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.894332 5367 scope.go:122] "RemoveContainer" containerID="dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b" Oct 29 22:45:25.959405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.959382 5367 scope.go:122] "RemoveContainer" containerID="dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b" Oct 29 22:45:25.959735 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:25.959679 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b\": container with ID starting with dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b not found: ID does not exist" containerID="dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b" Oct 29 22:45:25.959808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.959742 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b"} err="failed to get container status \"dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b\": rpc error: code = NotFound desc = could not find container \"dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b\": container with ID starting with dc98260c96125c4a4c15d4b3b9d9e24d01046eb5c7107c03c26940065e8fad3b not found: ID does not exist" Oct 29 22:45:25.959808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.959766 5367 scope.go:122] "RemoveContainer" containerID="07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf" Oct 29 22:45:25.971470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.971446 5367 scope.go:122] "RemoveContainer" containerID="07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf" Oct 29 22:45:25.971738 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:25.971711 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf\": container with ID starting with 07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf not found: ID does not exist" containerID="07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf" Oct 29 22:45:25.971833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.971750 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf"} err="failed to get container status \"07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf\": rpc error: code = NotFound desc = could not find container \"07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf\": container with ID starting with 07f30022b8cecc34e3b58ea2b9a5dec6413d9f16faca329958ae932147b68bcf not found: ID does not exist" Oct 29 22:45:25.971833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.971774 5367 scope.go:122] "RemoveContainer" containerID="aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc" Oct 29 22:45:25.982296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.982277 5367 scope.go:122] "RemoveContainer" containerID="aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc" Oct 29 22:45:25.982596 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:25.982571 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc\": container with ID starting with aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc not found: ID does not exist" containerID="aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc" Oct 29 22:45:25.982690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.982606 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc"} err="failed to get container status \"aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc\": rpc error: code = NotFound desc = could not find container \"aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc\": container with ID starting with aa10d5335a2dc7d2c0fecdd5b6f3d22973b4547cbad99d3087f1ea34832260cc not found: ID does not exist" Oct 29 22:45:25.982690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.982624 5367 scope.go:122] "RemoveContainer" containerID="6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c" Oct 29 22:45:25.993896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.993692 5367 scope.go:122] "RemoveContainer" containerID="6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c" Oct 29 22:45:25.994000 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:25.993980 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c\": container with ID starting with 6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c not found: ID does not exist" containerID="6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c" Oct 29 22:45:25.994043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.994010 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c"} err="failed to get container status \"6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c\": rpc error: code = NotFound desc = could not find container \"6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c\": container with ID starting with 6412c6daa70b653270da6ba9977b1e6ee3dc37f664681cc84d63455d7ee0148c not found: ID does not exist" Oct 29 22:45:25.994043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:25.994028 5367 scope.go:122] "RemoveContainer" containerID="000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108" Oct 29 22:45:26.003803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.003774 5367 scope.go:122] "RemoveContainer" containerID="9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2" Oct 29 22:45:26.013992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.013973 5367 scope.go:122] "RemoveContainer" containerID="000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108" Oct 29 22:45:26.014275 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:26.014255 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108\": container with ID starting with 000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108 not found: ID does not exist" containerID="000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108" Oct 29 22:45:26.014326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.014282 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108"} err="failed to get container status \"000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108\": rpc error: code = NotFound desc = could not find container \"000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108\": container with ID starting with 000d1a7dba0edc770eb476c7dee15500692702b9638018e3866778103ca10108 not found: ID does not exist" Oct 29 22:45:26.014326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.014299 5367 scope.go:122] "RemoveContainer" containerID="9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2" Oct 29 22:45:26.014783 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:26.014608 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2\": container with ID starting with 9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2 not found: ID does not exist" containerID="9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2" Oct 29 22:45:26.014783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.014646 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2"} err="failed to get container status \"9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2\": rpc error: code = NotFound desc = could not find container \"9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2\": container with ID starting with 9ee343e95c45e483aab277cebcfba634839db4865856027e3bcf033127a178b2 not found: ID does not exist" Oct 29 22:45:26.017067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.017044 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd"] Oct 29 22:45:26.021839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.021820 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-5smjd"] Oct 29 22:45:26.056226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.056163 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl"] Oct 29 22:45:26.062062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.062038 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-bn8dl"] Oct 29 22:45:26.097279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.097248 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86"] Oct 29 22:45:26.102155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.102129 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-xkm86"] Oct 29 22:45:26.138822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.138791 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x"] Oct 29 22:45:26.143441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.143416 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-4lc2x"] Oct 29 22:45:26.178428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.178389 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k"] Oct 29 22:45:26.188297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.188268 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-k928k"] Oct 29 22:45:26.245277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.245208 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n"] Oct 29 22:45:26.265474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.265441 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n"] Oct 29 22:45:26.339641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.339608 5367 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:45:26.340052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.339672 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-88b8n" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:26.608038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.608010 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:45:26.629920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.629890 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt"] Oct 29 22:45:26.630649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630634 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630651 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630660 5367 cpu_manager.go:414] "containerMap: removing container" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630665 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630678 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630683 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630709 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630714 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" Oct 29 22:45:26.630720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630723 5367 cpu_manager.go:414] "containerMap: removing container" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630729 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630739 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630743 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630754 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630758 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630764 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630769 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630787 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630791 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630801 5367 cpu_manager.go:414] "containerMap: removing container" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630806 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630823 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630827 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630834 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630839 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630945 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630954 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630966 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71d1651-adbf-4921-80f8-a616e5cb4b7d" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630974 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630979 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e05508d-5543-4f63-8303-369e5b276e93" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630985 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7115d2c4-0644-498c-9e4d-227f4d278139" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630992 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="757a1250-1588-47cd-9de6-46b0a0958a6e" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.630999 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f992356-17a9-4281-943f-a567571fd9ef" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.631005 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.631013 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.631018 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" containerName="istio-proxy" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.631025 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7146a9-1a97-4446-b808-1d218616cf30" containerName="cleanup" Oct 29 22:45:26.631024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.631037 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" containerName="istio-proxy" Oct 29 22:45:26.632258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.631045 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ba218c-89fd-4a1f-98a0-452f5e325e8b" containerName="cleanup" Oct 29 22:45:26.634837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.634821 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.667162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.667131 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt"] Oct 29 22:45:26.692693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.692660 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.692959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.692794 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.692959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.692875 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.692959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.692909 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.693123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.692971 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.693123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693018 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.693229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693162 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:26.693229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693171 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.693334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693249 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.693391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693344 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo\") pod \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\" (UID: \"5ee2d684-e4f5-42cb-91fb-0ec0c893c219\") " Oct 29 22:45:26.693447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693422 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693772 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.693881 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694023 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m8hq\" (UniqueName: \"kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694118 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694322 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694479 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694608 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694625 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694638 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694652 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.695204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.694665 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.703497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.697150 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:26.703497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.703155 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:26.703768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.703737 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "kube-api-access-gsdlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:26.703854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.703833 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy" pod "5ee2d684-e4f5-42cb-91fb-0ec0c893c219" (UID: "5ee2d684-e4f5-42cb-91fb-0ec0c893c219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.796392 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796292 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m8hq\" (UniqueName: \"kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.796569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796496 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.796645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796611 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.796790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796775 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.796858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796794 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdlq\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-kube-api-access-gsdlq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.796858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796807 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.796858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.796820 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2d684-e4f5-42cb-91fb-0ec0c893c219-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.797386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.797362 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.798844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.798825 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.816235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.816211 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m8hq\" (UniqueName: \"kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq\") pod \"maas-api-revoke-sub-1f870ae3-ppmwt\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.908959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.908926 5367 generic.go:356] "Generic (PLEG): container finished" podID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" containerID="6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993" exitCode=0 Oct 29 22:45:26.909121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.908989 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" Oct 29 22:45:26.909121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.909016 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" event={"ID":"5ee2d684-e4f5-42cb-91fb-0ec0c893c219","Type":"ContainerDied","Data":"6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993"} Oct 29 22:45:26.909121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.909071 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc" event={"ID":"5ee2d684-e4f5-42cb-91fb-0ec0c893c219","Type":"ContainerDied","Data":"fda1a2716f3aa3b3efcf4657ac7ac91ac142a7845d64c92892cfa2f435bc94f2"} Oct 29 22:45:26.909121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.909094 5367 scope.go:122] "RemoveContainer" containerID="6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993" Oct 29 22:45:26.915188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.915163 5367 generic.go:356] "Generic (PLEG): container finished" podID="1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" containerID="51362b4af5fde51902ce1a1217b00f6f21b3e41d9a5cceba70f3636a70563396" exitCode=0 Oct 29 22:45:26.915301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.915276 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" event={"ID":"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb","Type":"ContainerDied","Data":"51362b4af5fde51902ce1a1217b00f6f21b3e41d9a5cceba70f3636a70563396"} Oct 29 22:45:26.926495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.926474 5367 scope.go:122] "RemoveContainer" containerID="6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993" Oct 29 22:45:26.926805 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:26.926781 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993\": container with ID starting with 6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993 not found: ID does not exist" containerID="6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993" Oct 29 22:45:26.926898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.926810 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993"} err="failed to get container status \"6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993\": rpc error: code = NotFound desc = could not find container \"6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993\": container with ID starting with 6fdee958be9f14e62651121285ab496cfe30b8ec07d7c9cdd08f25b381b84993 not found: ID does not exist" Oct 29 22:45:26.949153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.949128 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:26.972751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.972723 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc"] Oct 29 22:45:26.976496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:26.976470 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ztfwc"] Oct 29 22:45:27.243478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.243421 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2667a74a-dee8-425f-87be-3d9a39bba51e" path="/var/lib/kubelet/pods/2667a74a-dee8-425f-87be-3d9a39bba51e/volumes" Oct 29 22:45:27.244668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.244642 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583be168-6bc5-4983-a98c-9bca968b4b97" path="/var/lib/kubelet/pods/583be168-6bc5-4983-a98c-9bca968b4b97/volumes" Oct 29 22:45:27.245714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.245683 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee2d684-e4f5-42cb-91fb-0ec0c893c219" path="/var/lib/kubelet/pods/5ee2d684-e4f5-42cb-91fb-0ec0c893c219/volumes" Oct 29 22:45:27.247089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.247072 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f992356-17a9-4281-943f-a567571fd9ef" path="/var/lib/kubelet/pods/5f992356-17a9-4281-943f-a567571fd9ef/volumes" Oct 29 22:45:27.247984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.247965 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc578fbe-af4d-42d1-8da6-169c2e314e1e" path="/var/lib/kubelet/pods/bc578fbe-af4d-42d1-8da6-169c2e314e1e/volumes" Oct 29 22:45:27.248983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.248953 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97d9d81-60f1-47f2-98fb-997c8c4de79d" path="/var/lib/kubelet/pods/d97d9d81-60f1-47f2-98fb-997c8c4de79d/volumes" Oct 29 22:45:27.249781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.249761 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51b751f-722b-4001-95cd-04ce87fd5356" path="/var/lib/kubelet/pods/f51b751f-722b-4001-95cd-04ce87fd5356/volumes" Oct 29 22:45:27.273858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.273828 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt"] Oct 29 22:45:27.274766 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:27.274738 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23650b7_03fa_49f7_b2f9_2e75757c7a63.slice/crio-c6e1ce0e12fb29f3b1f52edc264e8b611e040cd76bcaa02110bce154213e8e14 WatchSource:0}: Error finding container c6e1ce0e12fb29f3b1f52edc264e8b611e040cd76bcaa02110bce154213e8e14: Status 404 returned error can't find the container with id c6e1ce0e12fb29f3b1f52edc264e8b611e040cd76bcaa02110bce154213e8e14 Oct 29 22:45:27.933765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.933633 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" event={"ID":"e23650b7-03fa-49f7-b2f9-2e75757c7a63","Type":"ContainerStarted","Data":"f30ab670209a905ed88798f8aeac53329c94dde18a7dac03cf2bc02afc74be8f"} Oct 29 22:45:27.933765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.933716 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" event={"ID":"e23650b7-03fa-49f7-b2f9-2e75757c7a63","Type":"ContainerStarted","Data":"c6e1ce0e12fb29f3b1f52edc264e8b611e040cd76bcaa02110bce154213e8e14"} Oct 29 22:45:27.978822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:27.978772 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" podStartSLOduration=1.978759178 podStartE2EDuration="1.978759178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:27.972619908 +0000 UTC m=+1794.866694843" watchObservedRunningTime="2026-10-29 22:45:27.978759178 +0000 UTC m=+1794.872834098" Oct 29 22:45:28.115557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.115530 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:28.221423 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:28.221392 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23650b7_03fa_49f7_b2f9_2e75757c7a63.slice/crio-f30ab670209a905ed88798f8aeac53329c94dde18a7dac03cf2bc02afc74be8f.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:45:28.227801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.227775 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca\") pod \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " Oct 29 22:45:28.227974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.227844 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8\" (UniqueName: \"kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8\") pod \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " Oct 29 22:45:28.227974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.227903 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca\") pod \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\" (UID: \"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb\") " Oct 29 22:45:28.229774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.229751 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca" pod "1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" (UID: "1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:28.232485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.232456 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca" pod "1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" (UID: "1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:28.232602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.232583 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8" pod "1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" (UID: "1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb"). InnerVolumeSpecName "kube-api-access-fm5j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:28.330751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.330720 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:28.330751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.330748 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5j8\" (UniqueName: \"kubernetes.io/projected/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-kube-api-access-fm5j8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:28.330751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.330759 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:28.944792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.944757 5367 generic.go:356] "Generic (PLEG): container finished" podID="e23650b7-03fa-49f7-b2f9-2e75757c7a63" containerID="f30ab670209a905ed88798f8aeac53329c94dde18a7dac03cf2bc02afc74be8f" exitCode=0 Oct 29 22:45:28.945246 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.944863 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" event={"ID":"e23650b7-03fa-49f7-b2f9-2e75757c7a63","Type":"ContainerDied","Data":"f30ab670209a905ed88798f8aeac53329c94dde18a7dac03cf2bc02afc74be8f"} Oct 29 22:45:28.946771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.946743 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" Oct 29 22:45:28.947035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.946768 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt" event={"ID":"1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb","Type":"ContainerDied","Data":"7ec0ce19651e2aedaab334b98cdc97f89e8557afb1bb0f5141073a907a9b70f8"} Oct 29 22:45:28.947035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:28.946812 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec0ce19651e2aedaab334b98cdc97f89e8557afb1bb0f5141073a907a9b70f8" Oct 29 22:45:30.262260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.262223 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:30.406949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.406918 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca\") pod \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " Oct 29 22:45:30.407144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.407115 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq\" (UniqueName: \"kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq\") pod \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " Oct 29 22:45:30.407210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.407194 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca\") pod \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\" (UID: \"e23650b7-03fa-49f7-b2f9-2e75757c7a63\") " Oct 29 22:45:30.407249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.407216 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca" pod "e23650b7-03fa-49f7-b2f9-2e75757c7a63" (UID: "e23650b7-03fa-49f7-b2f9-2e75757c7a63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:30.407843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.407821 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:30.409183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.409152 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca" pod "e23650b7-03fa-49f7-b2f9-2e75757c7a63" (UID: "e23650b7-03fa-49f7-b2f9-2e75757c7a63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:30.409569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.409548 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq" pod "e23650b7-03fa-49f7-b2f9-2e75757c7a63" (UID: "e23650b7-03fa-49f7-b2f9-2e75757c7a63"). InnerVolumeSpecName "kube-api-access-8m8hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:30.509332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.509303 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m8hq\" (UniqueName: \"kubernetes.io/projected/e23650b7-03fa-49f7-b2f9-2e75757c7a63-kube-api-access-8m8hq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:30.509332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:30.509328 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23650b7-03fa-49f7-b2f9-2e75757c7a63-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:31.032003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:31.031970 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" event={"ID":"e23650b7-03fa-49f7-b2f9-2e75757c7a63","Type":"ContainerDied","Data":"c6e1ce0e12fb29f3b1f52edc264e8b611e040cd76bcaa02110bce154213e8e14"} Oct 29 22:45:31.032158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:31.032022 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6e1ce0e12fb29f3b1f52edc264e8b611e040cd76bcaa02110bce154213e8e14" Oct 29 22:45:31.032158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:31.031982 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt" Oct 29 22:45:31.042762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:31.042737 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz"] Oct 29 22:45:31.050522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:31.050485 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0e46a-ckttz"] Oct 29 22:45:31.224853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:31.224825 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf889cf-00cf-4e06-bf6e-13d7583de082" path="/var/lib/kubelet/pods/8cf889cf-00cf-4e06-bf6e-13d7583de082/volumes" Oct 29 22:45:32.032226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:32.032190 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp"] Oct 29 22:45:32.044306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:32.044267 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2843a034-bktmp"] Oct 29 22:45:33.224592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:33.224560 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa54447a-c35c-4b53-b5bf-d6936a391ae9" path="/var/lib/kubelet/pods/fa54447a-c35c-4b53-b5bf-d6936a391ae9/volumes" Oct 29 22:45:35.168016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:35.167985 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn"] Oct 29 22:45:35.185751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:35.185729 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4"] Oct 29 22:45:38.950925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.950886 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz"] Oct 29 22:45:38.952063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.952038 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e23650b7-03fa-49f7-b2f9-2e75757c7a63" containerName="revoke-keys" Oct 29 22:45:38.952244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.952220 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23650b7-03fa-49f7-b2f9-2e75757c7a63" containerName="revoke-keys" Oct 29 22:45:38.952347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.952332 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" containerName="revoke-keys" Oct 29 22:45:38.952347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.952340 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" containerName="revoke-keys" Oct 29 22:45:38.952536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.952498 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" containerName="revoke-keys" Oct 29 22:45:38.952536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.952519 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23650b7-03fa-49f7-b2f9-2e75757c7a63" containerName="revoke-keys" Oct 29 22:45:38.959043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.959024 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:38.975668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.975642 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz"] Oct 29 22:45:38.981913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:38.981887 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ad69b4\"" Oct 29 22:45:39.035721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.022412 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.035721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.022497 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfj78\" (UniqueName: \"kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78\") pod \"payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.035721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.029077 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ad69b4-dockercfg-wh6lx\"" Oct 29 22:45:39.132229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.130095 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.132229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.130161 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfj78\" (UniqueName: \"kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78\") pod \"payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.132229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.131113 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.140938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.139589 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t"] Oct 29 22:45:39.146171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.146144 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.162603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.159908 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfj78\" (UniqueName: \"kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78\") pod \"payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.173929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.172372 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t"] Oct 29 22:45:39.256273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.238245 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5blv\" (UniqueName: \"kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv\") pod \"payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.256273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.238426 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.278641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.278600 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:39.339983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.339945 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.340435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.340151 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5blv\" (UniqueName: \"kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv\") pod \"payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.341215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.341165 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.363977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.363895 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5blv\" (UniqueName: \"kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv\") pod \"payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.530590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.530516 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:39.598709 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:39.597318 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc054d32_b8a2_4613_aeaf_37c1c98b0862.slice/crio-e711c0cd9aeeac763e0d5113b18bc3ee31c9fc40eb24f26de274b0381fbf9297 WatchSource:0}: Error finding container e711c0cd9aeeac763e0d5113b18bc3ee31c9fc40eb24f26de274b0381fbf9297: Status 404 returned error can't find the container with id e711c0cd9aeeac763e0d5113b18bc3ee31c9fc40eb24f26de274b0381fbf9297 Oct 29 22:45:39.598709 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.598624 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz"] Oct 29 22:45:39.839828 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:39.839800 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc928179_f6f2_4cc6_a4ad_c5653601e838.slice/crio-289fbde98de3892464ab3b11bf3084ddd478d5ce2838a98bb694810c1a1dd567 WatchSource:0}: Error finding container 289fbde98de3892464ab3b11bf3084ddd478d5ce2838a98bb694810c1a1dd567: Status 404 returned error can't find the container with id 289fbde98de3892464ab3b11bf3084ddd478d5ce2838a98bb694810c1a1dd567 Oct 29 22:45:39.841105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:39.841072 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t"] Oct 29 22:45:40.172776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.171805 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerName="payload-processing" containerID="cri-o://95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9" gracePeriod=25 Oct 29 22:45:40.195720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.188825 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerName="payload-pre-processing" containerID="cri-o://b372505febf57337b95d6a74986e9c60faef4ccefa94d8e35a619b6293f13c2b" gracePeriod=25 Oct 29 22:45:40.211097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.211059 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" event={"ID":"cc928179-f6f2-4cc6-a4ad-c5653601e838","Type":"ContainerStarted","Data":"ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0"} Oct 29 22:45:40.211269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.211114 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" event={"ID":"cc928179-f6f2-4cc6-a4ad-c5653601e838","Type":"ContainerStarted","Data":"289fbde98de3892464ab3b11bf3084ddd478d5ce2838a98bb694810c1a1dd567"} Oct 29 22:45:40.211269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.211146 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:40.215711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.215662 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" event={"ID":"dc054d32-b8a2-4613-aeaf-37c1c98b0862","Type":"ContainerStarted","Data":"299b16042c6e366a21bb358204a645fb4eabeb7a170bd6e107ddfede1c6d743a"} Oct 29 22:45:40.215845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.215732 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" event={"ID":"dc054d32-b8a2-4613-aeaf-37c1c98b0862","Type":"ContainerStarted","Data":"e711c0cd9aeeac763e0d5113b18bc3ee31c9fc40eb24f26de274b0381fbf9297"} Oct 29 22:45:40.215845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.215774 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:40.278318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.278255 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" podStartSLOduration=1.278237303 podStartE2EDuration="1.278237303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:40.260013434 +0000 UTC m=+1807.154088355" watchObservedRunningTime="2026-10-29 22:45:40.278237303 +0000 UTC m=+1807.172312225" Oct 29 22:45:40.309408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:40.309347 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" podStartSLOduration=2.309327396 podStartE2EDuration="2.309327396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:40.302560561 +0000 UTC m=+1807.196635481" watchObservedRunningTime="2026-10-29 22:45:40.309327396 +0000 UTC m=+1807.203402317" Oct 29 22:45:41.237548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:41.237514 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt"] Oct 29 22:45:41.252668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:41.252627 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a26426-6v2dt"] Oct 29 22:45:43.225085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:43.225052 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb" path="/var/lib/kubelet/pods/1dc3c6e4-5221-4bb1-97bf-ec0ed027abdb/volumes" Oct 29 22:45:47.196658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:47.196573 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:47.197437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:47.197407 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:48.228156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.228123 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm"] Oct 29 22:45:48.228725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.228349 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" containerID="cri-o://f234bf16f00f179fbc78421641f70fd7827ebf982082fe00073f24071f1d1e79" gracePeriod=30 Oct 29 22:45:48.714363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.714325 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv"] Oct 29 22:45:48.720692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.720670 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:48.790785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.785792 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv"] Oct 29 22:45:48.819719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.813227 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:45:48.895298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.891096 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:48.898993 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.898784 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhrt\" (UniqueName: \"kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:48.904026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.903586 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:48.904026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:48.903648 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.005953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.005862 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhrt\" (UniqueName: \"kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.006115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.006002 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.006115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.006033 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.006798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.006763 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.008456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.008433 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.028247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.028210 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhrt\" (UniqueName: \"kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt\") pod \"maas-api-revoke-keys-e2e-inf-b-e224cd-898jv\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.181584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.181540 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:49.505287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:49.505254 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv"] Oct 29 22:45:49.505891 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:45:49.505848 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff7526d_938c_40ec_8e34_942e9e7a08ae.slice/crio-ebe783bde808634efc07046216c3c30dcff548c03ac6f1c77a6243df9ba6030c WatchSource:0}: Error finding container ebe783bde808634efc07046216c3c30dcff548c03ac6f1c77a6243df9ba6030c: Status 404 returned error can't find the container with id ebe783bde808634efc07046216c3c30dcff548c03ac6f1c77a6243df9ba6030c Oct 29 22:45:50.472408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:50.472361 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" event={"ID":"1ff7526d-938c-40ec-8e34-942e9e7a08ae","Type":"ContainerStarted","Data":"f7a62dded2682b2675d9c410e807c9916dbad17bb2750b9ff209bf50ce274a7f"} Oct 29 22:45:50.472616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:50.472427 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" event={"ID":"1ff7526d-938c-40ec-8e34-942e9e7a08ae","Type":"ContainerStarted","Data":"ebe783bde808634efc07046216c3c30dcff548c03ac6f1c77a6243df9ba6030c"} Oct 29 22:45:50.517091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:50.517043 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" podStartSLOduration=2.517029394 podStartE2EDuration="2.517029394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:50.514295379 +0000 UTC m=+1817.408370299" watchObservedRunningTime="2026-10-29 22:45:50.517029394 +0000 UTC m=+1817.411104356" Oct 29 22:45:51.232727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:51.232470 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:45:51.232973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:51.232947 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:45:51.831611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:51.823765 5367 generic.go:356] "Generic (PLEG): container finished" podID="1ff7526d-938c-40ec-8e34-942e9e7a08ae" containerID="f7a62dded2682b2675d9c410e807c9916dbad17bb2750b9ff209bf50ce274a7f" exitCode=0 Oct 29 22:45:51.831611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:51.823970 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" event={"ID":"1ff7526d-938c-40ec-8e34-942e9e7a08ae","Type":"ContainerDied","Data":"f7a62dded2682b2675d9c410e807c9916dbad17bb2750b9ff209bf50ce274a7f"} Oct 29 22:45:52.532974 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:45:52.532931 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:45:52.572298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:52.571117 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:52.638725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:52.612867 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.167783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.130117 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4"] Oct 29 22:45:53.335970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.295105 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.431715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.431597 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:45:53.465724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.465572 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:45:53.523012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.465905 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.523012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.470060 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:45:53.523012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.486976 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.523012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.522971 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.587062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.577181 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.639723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.632251 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.726867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.715037 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.742721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.738841 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:53.933723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.932649 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.933723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.933580 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:45:53.933723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.933693 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4"] Oct 29 22:45:53.962722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962441 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.962722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962533 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z44z\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.962722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962582 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.962722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962657 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.962722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962686 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.963166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962769 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.963166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962809 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.963166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.962917 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:53.963166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:53.963087 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.013774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.012877 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a558b3-openshift-default-dockercfg-q7ghg\"" Oct 29 22:45:54.211335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.211305 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:45:54.336421 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.299209 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:54.336421 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.299981 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:45:54.400474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.398897 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.403031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.402882 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.419732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.419677 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.420132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.420107 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.420383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.420366 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z44z\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.420584 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.420553 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.420842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.420823 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.424810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.424769 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.425109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.425074 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.561790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.442224 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.563728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.443292 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.569275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.444401 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.570266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.508576 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.570452 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.534232 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:45:54.570862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.534306 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Oct 29 22:45:54.571498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.571476 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:45:54.572373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.562775 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:54.573032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.562877 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Oct 29 22:45:54.573368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.562906 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 29 22:45:54.574075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.574054 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:45:54.574240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.574226 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:45:54.574373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.574361 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:45:54.575199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.569050 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:45:54.577640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.577614 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:54.619083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.618991 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:54.644725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.619818 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:45:54.646087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.646049 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:54.651722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.648213 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.749722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.717244 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.749722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.718913 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z44z\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.749722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.735874 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:54.757518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.757481 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:54.816777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.770656 5367 generic.go:356] "Generic (PLEG): container finished" podID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerID="f234bf16f00f179fbc78421641f70fd7827ebf982082fe00073f24071f1d1e79" exitCode=0 Oct 29 22:45:54.816777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.770683 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" event={"ID":"722caee6-f913-4d87-a3b3-9342ccc42ebe","Type":"ContainerDied","Data":"f234bf16f00f179fbc78421641f70fd7827ebf982082fe00073f24071f1d1e79"} Oct 29 22:45:54.816777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.782278 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:45:54.816777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.782781 5367 scope.go:122] "RemoveContainer" containerID="4548dd87f251184c4db6b87237a44e023d49ea243c794942a2109c91512bb3c5" Oct 29 22:45:54.852389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.829542 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 22:45:54.928771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.920970 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:45:54.928771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.921540 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:45:54.979288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.929937 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 22:45:54.979288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:54.951456 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:45:55.148721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.126283 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:55.148965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.148303 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:55.184775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.163593 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:55.184775 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.171227 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 22:45:55.257250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.257217 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca\") pod \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " Oct 29 22:45:55.257882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.257856 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca\") pod \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " Oct 29 22:45:55.258207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.258173 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt\" (UniqueName: \"kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt\") pod \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\" (UID: \"1ff7526d-938c-40ec-8e34-942e9e7a08ae\") " Oct 29 22:45:55.325826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.284029 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca" pod "1ff7526d-938c-40ec-8e34-942e9e7a08ae" (UID: "1ff7526d-938c-40ec-8e34-942e9e7a08ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:55.329327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.329290 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" podUID="ba4c4883-c331-458f-8060-c35e65211fd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:55.367401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.367372 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.415724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.395136 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:45:55.420846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.417146 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt" pod "1ff7526d-938c-40ec-8e34-942e9e7a08ae" (UID: "1ff7526d-938c-40ec-8e34-942e9e7a08ae"). InnerVolumeSpecName "kube-api-access-qrhrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:55.420846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.420010 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca" pod "1ff7526d-938c-40ec-8e34-942e9e7a08ae" (UID: "1ff7526d-938c-40ec-8e34-942e9e7a08ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:55.420846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.420128 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 22:45:55.531136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.531104 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 22:45:55.540014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.539931 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrhrt\" (UniqueName: \"kubernetes.io/projected/1ff7526d-938c-40ec-8e34-942e9e7a08ae-kube-api-access-qrhrt\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.540014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.539965 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff7526d-938c-40ec-8e34-942e9e7a08ae-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.540782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.540498 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb" Oct 29 22:45:55.540782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.540552 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 22:45:55.545068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.544901 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 22:45:55.550752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.545619 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 22:45:55.641820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.641573 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.641820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.641635 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.641820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.641689 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.641820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.641762 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.643618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.642410 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.643618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.642543 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.643618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.642647 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.643618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.642763 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.643618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.642804 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55\") pod \"722caee6-f913-4d87-a3b3-9342ccc42ebe\" (UID: \"722caee6-f913-4d87-a3b3-9342ccc42ebe\") " Oct 29 22:45:55.645160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.644459 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.645160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.644682 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.645160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.644886 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.645714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.645454 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.647933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.645977 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:55.648868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.648847 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.649007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.648992 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722caee6-f913-4d87-a3b3-9342ccc42ebe-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.649109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.649098 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.649203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.649192 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.649308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.649295 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.651187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.651161 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.651323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.651305 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:55.651470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.651446 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:55.712582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.710135 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55" pod "722caee6-f913-4d87-a3b3-9342ccc42ebe" (UID: "722caee6-f913-4d87-a3b3-9342ccc42ebe"). InnerVolumeSpecName "kube-api-access-fws55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:55.771720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.762453 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.771720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.762499 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.771720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.762515 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722caee6-f913-4d87-a3b3-9342ccc42ebe-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.771720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.762529 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fws55\" (UniqueName: \"kubernetes.io/projected/722caee6-f913-4d87-a3b3-9342ccc42ebe-kube-api-access-fws55\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.846325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.846284 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" event={"ID":"722caee6-f913-4d87-a3b3-9342ccc42ebe","Type":"ContainerDied","Data":"9705aab95c12e646a8a371ce946a42cf91f1d83b0984d9eedb4bda7ed99e20c3"} Oct 29 22:45:55.846516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.846410 5367 scope.go:122] "RemoveContainer" containerID="f234bf16f00f179fbc78421641f70fd7827ebf982082fe00073f24071f1d1e79" Oct 29 22:45:55.847177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.846759 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm" Oct 29 22:45:55.853582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.853560 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" Oct 29 22:45:55.875035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.855503 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv" event={"ID":"1ff7526d-938c-40ec-8e34-942e9e7a08ae","Type":"ContainerDied","Data":"ebe783bde808634efc07046216c3c30dcff548c03ac6f1c77a6243df9ba6030c"} Oct 29 22:45:55.875035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:55.855554 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe783bde808634efc07046216c3c30dcff548c03ac6f1c77a6243df9ba6030c" Oct 29 22:45:56.252536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.252425 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4"] Oct 29 22:45:56.276728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.259453 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:45:56.276728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.259547 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:45:56.276728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.259591 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:45:56.750792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.750722 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm"] Oct 29 22:45:56.803835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.761635 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a26426-openshift-default-6766c9bbf-w6fnm"] Oct 29 22:45:56.917302 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.917251 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" event={"ID":"90549b69-0a71-4e73-9d2c-f9792b2aa082","Type":"ContainerStarted","Data":"cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873"} Oct 29 22:45:56.917509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.917322 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" event={"ID":"90549b69-0a71-4e73-9d2c-f9792b2aa082","Type":"ContainerStarted","Data":"f07fea3161fd58f2a153ff24fdab52484c0afa7be2b42eace4f759cb48ae8f00"} Oct 29 22:45:56.998723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:56.997955 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podStartSLOduration=4.99793436 podStartE2EDuration="4.99793436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:56.971022439 +0000 UTC m=+1823.865097361" watchObservedRunningTime="2026-10-29 22:45:56.99793436 +0000 UTC m=+1823.892009281" Oct 29 22:45:57.174553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:57.174468 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:45:57.174936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:57.174910 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:45:57.175174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:57.175148 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:45:57.205842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:57.205800 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:57.206186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:57.206163 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:57.230169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:57.228757 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" path="/var/lib/kubelet/pods/722caee6-f913-4d87-a3b3-9342ccc42ebe/volumes" Oct 29 22:45:58.184725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:58.167579 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:45:58.184725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:58.167649 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:45:59.179723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.175307 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:45:59.179723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.175380 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:45:59.603780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.603288 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624"] Oct 29 22:45:59.605554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.605511 5367 cpu_manager.go:414] "containerMap: removing container" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" Oct 29 22:45:59.605722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.605690 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" Oct 29 22:45:59.605845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.605833 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1ff7526d-938c-40ec-8e34-942e9e7a08ae" containerName="revoke-keys" Oct 29 22:45:59.605929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.605916 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff7526d-938c-40ec-8e34-942e9e7a08ae" containerName="revoke-keys" Oct 29 22:45:59.606051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.606040 5367 cpu_manager.go:414] "containerMap: removing container" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" Oct 29 22:45:59.606133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.606123 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" Oct 29 22:45:59.606440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.606421 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" Oct 29 22:45:59.606558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.606546 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff7526d-938c-40ec-8e34-942e9e7a08ae" containerName="revoke-keys" Oct 29 22:45:59.613253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.613220 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:45:59.628525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.628455 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624"] Oct 29 22:45:59.629511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.629224 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8ebc60-openshift-default-dockercfg-tvk92\"" Oct 29 22:45:59.797478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.796133 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4"] Oct 29 22:45:59.797478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.796997 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" podUID="58e1e05f-1de1-46fd-b66f-d376d29692b1" containerName="maas-api" containerID="cri-o://aad58c03db146a96830c3ca394339382b560175ff8d3b3d1503397725d79c637" gracePeriod=30 Oct 29 22:46:00.202132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:45:59.812965 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.202545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.202519 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.204604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.204581 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.205053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.205029 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.205468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.205451 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.205677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.205662 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.205885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.205852 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.205996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.191201 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:46:00.206161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.206137 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:46:00.206298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.206158 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdfk\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.206484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.206469 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.308155 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.308263 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.308731 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.308838 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.308908 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.308933 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.309292 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdfk\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.309417 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.309661 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.325667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.310885 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.342372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.342325 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.361739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.349988 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.371100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.370984 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.371418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.371398 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.371867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.371841 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.415006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.414922 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.417742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.417670 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdfk\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.418560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.417960 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:00.575723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:00.555075 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:01.070805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.070377 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:46:01.279018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.220669 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:46:01.279018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.220788 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:46:01.407209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.407170 5367 generic.go:356] "Generic (PLEG): container finished" podID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerID="b372505febf57337b95d6a74986e9c60faef4ccefa94d8e35a619b6293f13c2b" exitCode=0 Oct 29 22:46:01.407560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.407530 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" event={"ID":"358770a0-e2ae-42ef-a765-a96e610de1b0","Type":"ContainerDied","Data":"b372505febf57337b95d6a74986e9c60faef4ccefa94d8e35a619b6293f13c2b"} Oct 29 22:46:01.407771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.407750 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" event={"ID":"358770a0-e2ae-42ef-a765-a96e610de1b0","Type":"ContainerDied","Data":"86e702d0a43e12fa09e342404d2ae08e32dfc4b8d46fb366398e2cf9a0aaf117"} Oct 29 22:46:01.407908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.407894 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e702d0a43e12fa09e342404d2ae08e32dfc4b8d46fb366398e2cf9a0aaf117" Oct 29 22:46:01.409252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.409229 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume\") pod \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " Oct 29 22:46:01.410959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.410830 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp\" (UniqueName: \"kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp\") pod \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\" (UID: \"8e6c03f3-a7ad-4e26-9333-2668607d1ff1\") " Oct 29 22:46:01.412229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.412198 5367 generic.go:356] "Generic (PLEG): container finished" podID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerID="95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9" exitCode=0 Oct 29 22:46:01.412464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.412439 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" event={"ID":"8e6c03f3-a7ad-4e26-9333-2668607d1ff1","Type":"ContainerDied","Data":"95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9"} Oct 29 22:46:01.412551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.412496 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" event={"ID":"8e6c03f3-a7ad-4e26-9333-2668607d1ff1","Type":"ContainerDied","Data":"a2cf5b04602d1556ccbcab13ffabac10afc48efc8047fe3287580ef7fb34b329"} Oct 29 22:46:01.412551 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.412525 5367 scope.go:122] "RemoveContainer" containerID="95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9" Oct 29 22:46:01.413344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.412825 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn" Oct 29 22:46:01.414933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.414669 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume" pod "8e6c03f3-a7ad-4e26-9333-2668607d1ff1" (UID: "8e6c03f3-a7ad-4e26-9333-2668607d1ff1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:01.424984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.420298 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp" pod "8e6c03f3-a7ad-4e26-9333-2668607d1ff1" (UID: "8e6c03f3-a7ad-4e26-9333-2668607d1ff1"). InnerVolumeSpecName "kube-api-access-jdbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:01.424984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.423170 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:46:01.427174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.426150 5367 generic.go:356] "Generic (PLEG): container finished" podID="58e1e05f-1de1-46fd-b66f-d376d29692b1" containerID="aad58c03db146a96830c3ca394339382b560175ff8d3b3d1503397725d79c637" exitCode=0 Oct 29 22:46:01.427466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.427396 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" event={"ID":"58e1e05f-1de1-46fd-b66f-d376d29692b1","Type":"ContainerDied","Data":"aad58c03db146a96830c3ca394339382b560175ff8d3b3d1503397725d79c637"} Oct 29 22:46:01.431800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.431775 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:46:01.439328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.439185 5367 scope.go:122] "RemoveContainer" containerID="95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9" Oct 29 22:46:01.441099 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:01.441078 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9\": container with ID starting with 95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9 not found: ID does not exist" containerID="95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9" Oct 29 22:46:01.441286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.441264 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9"} err="failed to get container status \"95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9\": rpc error: code = NotFound desc = could not find container \"95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9\": container with ID starting with 95d9e62375346bdf3957f729590bef1c65e503884823fb0738ea93ebec8e40b9 not found: ID does not exist" Oct 29 22:46:01.519248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.519156 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.519248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.519224 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbcp\" (UniqueName: \"kubernetes.io/projected/8e6c03f3-a7ad-4e26-9333-2668607d1ff1-kube-api-access-jdbcp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.528813 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:01.528763 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08200b1_1dcf_440c_b1ef_6a66ba84e01a.slice/crio-2aea0dce0220edd04c40c5c3180a2abfc38a919dbcb05e0b06efea201d3cc442 WatchSource:0}: Error finding container 2aea0dce0220edd04c40c5c3180a2abfc38a919dbcb05e0b06efea201d3cc442: Status 404 returned error can't find the container with id 2aea0dce0220edd04c40c5c3180a2abfc38a919dbcb05e0b06efea201d3cc442 Oct 29 22:46:01.536886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.535146 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:01.536886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.535275 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:01.536886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.535317 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:01.536886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.536041 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624"] Oct 29 22:46:01.620404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.620372 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs\") pod \"58e1e05f-1de1-46fd-b66f-d376d29692b1\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " Oct 29 22:46:01.621086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.621047 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume" pod "358770a0-e2ae-42ef-a765-a96e610de1b0" (UID: "358770a0-e2ae-42ef-a765-a96e610de1b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:01.621644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.620688 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume\") pod \"358770a0-e2ae-42ef-a765-a96e610de1b0\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " Oct 29 22:46:01.621872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.621857 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b\" (UniqueName: \"kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b\") pod \"358770a0-e2ae-42ef-a765-a96e610de1b0\" (UID: \"358770a0-e2ae-42ef-a765-a96e610de1b0\") " Oct 29 22:46:01.622292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.622276 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls\") pod \"58e1e05f-1de1-46fd-b66f-d376d29692b1\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " Oct 29 22:46:01.622826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.622809 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz\" (UniqueName: \"kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz\") pod \"58e1e05f-1de1-46fd-b66f-d376d29692b1\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " Oct 29 22:46:01.623528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.623510 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca\") pod \"58e1e05f-1de1-46fd-b66f-d376d29692b1\" (UID: \"58e1e05f-1de1-46fd-b66f-d376d29692b1\") " Oct 29 22:46:01.624333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.624045 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca" pod "58e1e05f-1de1-46fd-b66f-d376d29692b1" (UID: "58e1e05f-1de1-46fd-b66f-d376d29692b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:01.625418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.625397 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358770a0-e2ae-42ef-a765-a96e610de1b0-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.625583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.625567 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1e05f-1de1-46fd-b66f-d376d29692b1-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.627186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.627152 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b" pod "358770a0-e2ae-42ef-a765-a96e610de1b0" (UID: "358770a0-e2ae-42ef-a765-a96e610de1b0"). InnerVolumeSpecName "kube-api-access-8ct7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:01.630569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.630522 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz" pod "58e1e05f-1de1-46fd-b66f-d376d29692b1" (UID: "58e1e05f-1de1-46fd-b66f-d376d29692b1"). InnerVolumeSpecName "kube-api-access-mfnvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:01.632138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.632096 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls" pod "58e1e05f-1de1-46fd-b66f-d376d29692b1" (UID: "58e1e05f-1de1-46fd-b66f-d376d29692b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:01.638109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.638072 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs" pod "58e1e05f-1de1-46fd-b66f-d376d29692b1" (UID: "58e1e05f-1de1-46fd-b66f-d376d29692b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:01.727343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.727312 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.727343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.727344 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ct7b\" (UniqueName: \"kubernetes.io/projected/358770a0-e2ae-42ef-a765-a96e610de1b0-kube-api-access-8ct7b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.727557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.727357 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e1e05f-1de1-46fd-b66f-d376d29692b1-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.727557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.727371 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfnvz\" (UniqueName: \"kubernetes.io/projected/58e1e05f-1de1-46fd-b66f-d376d29692b1-kube-api-access-mfnvz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:01.814016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.813937 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn"] Oct 29 22:46:01.820557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:01.820273 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a26426-6b9b8c6db6-rfhzn"] Oct 29 22:46:02.164816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.164324 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:46:02.164816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.164384 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:46:02.440770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.440041 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" event={"ID":"e08200b1-1dcf-440c-b1ef-6a66ba84e01a","Type":"ContainerStarted","Data":"201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99"} Oct 29 22:46:02.440770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.440101 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" event={"ID":"e08200b1-1dcf-440c-b1ef-6a66ba84e01a","Type":"ContainerStarted","Data":"2aea0dce0220edd04c40c5c3180a2abfc38a919dbcb05e0b06efea201d3cc442"} Oct 29 22:46:02.441458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.441431 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" Oct 29 22:46:02.441458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.441446 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4" event={"ID":"58e1e05f-1de1-46fd-b66f-d376d29692b1","Type":"ContainerDied","Data":"308cf2126c10f45d25537818c218ab1f5518f4109d43a59c3256c8ca8cfddf7b"} Oct 29 22:46:02.441668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.441507 5367 scope.go:122] "RemoveContainer" containerID="aad58c03db146a96830c3ca394339382b560175ff8d3b3d1503397725d79c637" Oct 29 22:46:02.441668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.441434 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4" Oct 29 22:46:02.503086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.503029 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podStartSLOduration=3.503012532 podStartE2EDuration="3.503012532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:02.500746387 +0000 UTC m=+1829.394821308" watchObservedRunningTime="2026-10-29 22:46:02.503012532 +0000 UTC m=+1829.397087452" Oct 29 22:46:02.572103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.571085 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:02.578489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.578424 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:02.594180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.594110 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4"] Oct 29 22:46:02.600623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.600582 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e224cd-7b48fbb9bd-cwjr4"] Oct 29 22:46:02.605146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.605113 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4"] Oct 29 22:46:02.609425 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.609393 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a26426-6dbb7f4b7-5fmf4"] Oct 29 22:46:02.619292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.619262 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv"] Oct 29 22:46:02.621037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.620795 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerName="payload-processing" Oct 29 22:46:02.621187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621074 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerName="payload-processing" Oct 29 22:46:02.621187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621160 5367 cpu_manager.go:414] "containerMap: removing container" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerName="payload-pre-processing" Oct 29 22:46:02.621187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621170 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerName="payload-pre-processing" Oct 29 22:46:02.621187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621182 5367 cpu_manager.go:414] "containerMap: removing container" podUID="58e1e05f-1de1-46fd-b66f-d376d29692b1" containerName="maas-api" Oct 29 22:46:02.621419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621191 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e1e05f-1de1-46fd-b66f-d376d29692b1" containerName="maas-api" Oct 29 22:46:02.621419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621394 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" containerName="payload-processing" Oct 29 22:46:02.621520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621421 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="722caee6-f913-4d87-a3b3-9342ccc42ebe" containerName="istio-proxy" Oct 29 22:46:02.621520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621429 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" containerName="payload-pre-processing" Oct 29 22:46:02.621520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.621440 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e1e05f-1de1-46fd-b66f-d376d29692b1" containerName="maas-api" Oct 29 22:46:02.747297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.746386 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv"] Oct 29 22:46:02.747297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.746543 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.807743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.806735 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8dd4c8-openshift-default-dockercfg-b8qn9\"" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.871922 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.871970 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.871998 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.872090 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljv4\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.872222 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.872271 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.872296 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.872452 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:02.873722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:02.872516 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007047 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007132 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007201 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007227 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007251 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007355 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljv4\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007504 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007559 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.007587 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.039819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.028780 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.043465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.042854 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.043465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.043123 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.043465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.043362 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.043923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.043880 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.101497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.101376 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.102694 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.102249 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.106305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.105396 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljv4\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.107090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.107067 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token\") pod \"e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.184562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.184341 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:03.185072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.184683 5367 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:46:03.185072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.184748 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:46:03.243736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.243649 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358770a0-e2ae-42ef-a765-a96e610de1b0" path="/var/lib/kubelet/pods/358770a0-e2ae-42ef-a765-a96e610de1b0/volumes" Oct 29 22:46:03.244455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.244424 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e1e05f-1de1-46fd-b66f-d376d29692b1" path="/var/lib/kubelet/pods/58e1e05f-1de1-46fd-b66f-d376d29692b1/volumes" Oct 29 22:46:03.245233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.245212 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6c03f3-a7ad-4e26-9333-2668607d1ff1" path="/var/lib/kubelet/pods/8e6c03f3-a7ad-4e26-9333-2668607d1ff1/volumes" Oct 29 22:46:03.453188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.453155 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:03.454138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.454112 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:46:03.591894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.591860 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv"] Oct 29 22:46:03.594265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.594231 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:03.594369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.594311 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:03.594369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:03.594356 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:04.192872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:04.192827 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:46:04.193924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:04.193899 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:46:04.194750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:04.194695 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:46:04.464904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:04.464822 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" event={"ID":"c2441bc5-e772-4c00-aca4-5a6dee3314e7","Type":"ContainerStarted","Data":"e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2"} Oct 29 22:46:04.464904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:04.464883 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" event={"ID":"c2441bc5-e772-4c00-aca4-5a6dee3314e7","Type":"ContainerStarted","Data":"1a44934879704eb1d341386473546fd020a21bb72cd4d4913087c81882f3fbce"} Oct 29 22:46:04.508565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:04.508123 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podStartSLOduration=2.5081099719999997 podStartE2EDuration="2.508109972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:04.507531204 +0000 UTC m=+1831.401606123" watchObservedRunningTime="2026-10-29 22:46:04.508109972 +0000 UTC m=+1831.402184892" Oct 29 22:46:05.185353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:05.185272 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:05.378454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:05.191517 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:05.476333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:05.476308 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:05.477599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:05.477561 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:46:06.486952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.486918 5367 generic.go:356] "Generic (PLEG): container finished" podID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerID="7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560" exitCode=0 Oct 29 22:46:06.487358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.486987 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" event={"ID":"2892e72b-9eb9-4ffa-abfa-a45306280418","Type":"ContainerDied","Data":"7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560"} Oct 29 22:46:06.487499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.487482 5367 scope.go:122] "RemoveContainer" containerID="7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560" Oct 29 22:46:06.488914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.488892 5367 generic.go:356] "Generic (PLEG): container finished" podID="e858318b-4888-4949-8200-ccc0e1719ec6" containerID="cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938" exitCode=0 Oct 29 22:46:06.488999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.488960 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" event={"ID":"e858318b-4888-4949-8200-ccc0e1719ec6","Type":"ContainerDied","Data":"cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938"} Oct 29 22:46:06.489401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.489384 5367 scope.go:122] "RemoveContainer" containerID="cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938" Oct 29 22:46:06.489447 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.489403 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:06.489492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.489466 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:06.489531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.489497 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:06.491150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.491125 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:06.491268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.491184 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:06.491268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.491220 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:06.936549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.935976 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq"] Oct 29 22:46:06.941059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.940961 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:06.965542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.965508 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8ebc60\"" Oct 29 22:46:06.965860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.965835 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8ebc60-dockercfg-2lfvf\"" Oct 29 22:46:06.973804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:06.973739 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq"] Oct 29 22:46:07.035723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.035396 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv"] Oct 29 22:46:07.048756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.045487 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.058572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.058538 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv"] Oct 29 22:46:07.165720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.151968 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9zn\" (UniqueName: \"kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn\") pod \"payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.165720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.152338 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.279810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.273053 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.279810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.273225 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9zn\" (UniqueName: \"kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn\") pod \"payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.279810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.273389 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wlb\" (UniqueName: \"kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb\") pod \"payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.279810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.273442 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.288256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.288209 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.327103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.327049 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9zn\" (UniqueName: \"kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn\") pod \"payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.375220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.374937 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wlb\" (UniqueName: \"kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb\") pod \"payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.375220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.375015 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.376068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.376036 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.395639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.395550 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wlb\" (UniqueName: \"kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb\") pod \"payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:07.506061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.505757 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" event={"ID":"2892e72b-9eb9-4ffa-abfa-a45306280418","Type":"ContainerStarted","Data":"66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727"} Oct 29 22:46:07.508839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.508806 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" event={"ID":"e858318b-4888-4949-8200-ccc0e1719ec6","Type":"ContainerStarted","Data":"9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69"} Oct 29 22:46:07.576269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.575981 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:07.664045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:07.663998 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:08.020466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.019338 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:46:08.020466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.019401 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:46:08.080382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.080091 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:46:08.080382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.080157 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 29 22:46:08.080636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.080462 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:46:08.080636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.080498 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 22:46:08.247194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.247159 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq"] Oct 29 22:46:08.313323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.313289 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv"] Oct 29 22:46:08.537617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.536289 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" event={"ID":"61c670f9-76f0-4d6a-a9a2-e2373fb09f83","Type":"ContainerStarted","Data":"9409a4ebf0acf854a03009ea7ec72677a22af3efb3915aa467fb3cf0ba07918c"} Oct 29 22:46:08.537617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.536353 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" event={"ID":"61c670f9-76f0-4d6a-a9a2-e2373fb09f83","Type":"ContainerStarted","Data":"16fa148241bbea1e375c608c9a7d1071583da35fcdd3f66e87ee89932c6c33cd"} Oct 29 22:46:08.537617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.537552 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:08.548768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.547942 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" event={"ID":"98998b4d-6e7e-417c-b4a9-0998313882de","Type":"ContainerStarted","Data":"ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9"} Oct 29 22:46:08.548768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.547998 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:08.548768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.548022 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" event={"ID":"98998b4d-6e7e-417c-b4a9-0998313882de","Type":"ContainerStarted","Data":"4fb255555fd778304ed20475e24c090d2b29f0bae26a2148874717bc051ef719"} Oct 29 22:46:08.622106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.618444 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" podStartSLOduration=2.618419761 podStartE2EDuration="2.618419761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:08.603885574 +0000 UTC m=+1835.497960501" watchObservedRunningTime="2026-10-29 22:46:08.618419761 +0000 UTC m=+1835.512494679" Oct 29 22:46:08.648954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.648683 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" podStartSLOduration=2.648669183 podStartE2EDuration="2.648669183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:08.64818611 +0000 UTC m=+1835.542261030" watchObservedRunningTime="2026-10-29 22:46:08.648669183 +0000 UTC m=+1835.542744102" Oct 29 22:46:08.906780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.900358 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:46:08.990876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.978080 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:46:08.990876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:08.978143 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 22:46:09.015765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.009465 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:46:09.015765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.009518 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 29 22:46:09.048939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.048887 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx"] Oct 29 22:46:09.103640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.103605 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.127490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.125055 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8dd4c8\"" Oct 29 22:46:09.127490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.125831 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8dd4c8-dockercfg-8wlns\"" Oct 29 22:46:09.173957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.173845 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdh7c\" (UniqueName: \"kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c\") pod \"payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.212258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.212216 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.275722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.185786 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx"] Oct 29 22:46:09.381622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.381572 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdh7c\" (UniqueName: \"kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c\") pod \"payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.381843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.381777 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.417776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.411852 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.440531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.438581 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm"] Oct 29 22:46:09.446040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.444961 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdh7c\" (UniqueName: \"kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c\") pod \"payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.515724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.511044 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm"] Oct 29 22:46:09.515724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.512322 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.644608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.644315 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:09.728788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.654103 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.728788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.654267 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxnj\" (UniqueName: \"kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj\") pod \"payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.781726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.778291 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.781726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.778448 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxnj\" (UniqueName: \"kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj\") pod \"payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.781726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.780298 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.865131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.861511 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxnj\" (UniqueName: \"kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj\") pod \"payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:09.930105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.930067 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:10.004790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.999567 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.004790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:09.999628 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 22:46:10.005272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.005138 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.005272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.005215 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 29 22:46:10.235096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.232116 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8"] Oct 29 22:46:10.235096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.232375 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" podUID="0e712137-2012-4419-90e9-95236293218a" containerName="maas-api" containerID="cri-o://9924244fee9753ea90eac4cde6c3be00f9d602dcdfcdc0ab05238205daf564e8" gracePeriod=30 Oct 29 22:46:10.307872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.307824 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" podUID="0e712137-2012-4419-90e9-95236293218a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.194:8443/health\": dial tcp 10.133.0.194:8443: connect: connection refused" Oct 29 22:46:10.316951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.316768 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx"] Oct 29 22:46:10.318106 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:10.318076 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bfe2cc_0407_4452_ac95_a758aec35d52.slice/crio-9337556ef05d524700aa91860b00c55d48e183bc7e8a4631301335339a00adc0 WatchSource:0}: Error finding container 9337556ef05d524700aa91860b00c55d48e183bc7e8a4631301335339a00adc0: Status 404 returned error can't find the container with id 9337556ef05d524700aa91860b00c55d48e183bc7e8a4631301335339a00adc0 Oct 29 22:46:10.687024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.679716 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm"] Oct 29 22:46:10.713766 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:10.713367 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859b88c4_18d2_4c33_936d_b5b20a2a32f0.slice/crio-09c03f7e3549e77aaa9df25ac656efd0945ce1c764989e0e11d3c41aa6f41f2b WatchSource:0}: Error finding container 09c03f7e3549e77aaa9df25ac656efd0945ce1c764989e0e11d3c41aa6f41f2b: Status 404 returned error can't find the container with id 09c03f7e3549e77aaa9df25ac656efd0945ce1c764989e0e11d3c41aa6f41f2b Oct 29 22:46:10.986600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.986511 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.986925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.986892 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 22:46:10.987685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.987651 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.988002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.987979 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 29 22:46:10.988121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:10.987938 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:46:11.115904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.110038 5367 generic.go:356] "Generic (PLEG): container finished" podID="0e712137-2012-4419-90e9-95236293218a" containerID="9924244fee9753ea90eac4cde6c3be00f9d602dcdfcdc0ab05238205daf564e8" exitCode=0 Oct 29 22:46:11.131144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.130388 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" event={"ID":"0e712137-2012-4419-90e9-95236293218a","Type":"ContainerDied","Data":"9924244fee9753ea90eac4cde6c3be00f9d602dcdfcdc0ab05238205daf564e8"} Oct 29 22:46:11.221729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.210814 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" event={"ID":"64bfe2cc-0407-4452-ac95-a758aec35d52","Type":"ContainerStarted","Data":"9337556ef05d524700aa91860b00c55d48e183bc7e8a4631301335339a00adc0"} Oct 29 22:46:11.457903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.457871 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:46:11.566856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.563322 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96\" (UniqueName: \"kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96\") pod \"0e712137-2012-4419-90e9-95236293218a\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " Oct 29 22:46:11.566856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.563530 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca\") pod \"0e712137-2012-4419-90e9-95236293218a\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " Oct 29 22:46:11.566856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.563594 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls\") pod \"0e712137-2012-4419-90e9-95236293218a\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " Oct 29 22:46:11.566856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.563684 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs\") pod \"0e712137-2012-4419-90e9-95236293218a\" (UID: \"0e712137-2012-4419-90e9-95236293218a\") " Oct 29 22:46:11.604738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.577147 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca" pod "0e712137-2012-4419-90e9-95236293218a" (UID: "0e712137-2012-4419-90e9-95236293218a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:11.689722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.671425 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e712137-2012-4419-90e9-95236293218a-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:11.711734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.710062 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls" pod "0e712137-2012-4419-90e9-95236293218a" (UID: "0e712137-2012-4419-90e9-95236293218a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:11.739303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.739262 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs" pod "0e712137-2012-4419-90e9-95236293218a" (UID: "0e712137-2012-4419-90e9-95236293218a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:11.741853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.741826 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96" pod "0e712137-2012-4419-90e9-95236293218a" (UID: "0e712137-2012-4419-90e9-95236293218a"). InnerVolumeSpecName "kube-api-access-rmb96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:11.823766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.823729 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmb96\" (UniqueName: \"kubernetes.io/projected/0e712137-2012-4419-90e9-95236293218a-kube-api-access-rmb96\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:11.823997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.823980 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:11.824147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:11.824135 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e712137-2012-4419-90e9-95236293218a-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:12.000326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.000290 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.000619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.000587 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 22:46:12.000898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.000797 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.000898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.000846 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 29 22:46:12.213895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.213856 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7"] Oct 29 22:46:12.225145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.225109 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" event={"ID":"859b88c4-18d2-4c33-936d-b5b20a2a32f0","Type":"ContainerStarted","Data":"b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d"} Oct 29 22:46:12.225346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.225176 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:12.225346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.225200 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" event={"ID":"859b88c4-18d2-4c33-936d-b5b20a2a32f0","Type":"ContainerStarted","Data":"09c03f7e3549e77aaa9df25ac656efd0945ce1c764989e0e11d3c41aa6f41f2b"} Oct 29 22:46:12.226970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.226943 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63db1a-mhrn7"] Oct 29 22:46:12.227131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.227108 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" event={"ID":"64bfe2cc-0407-4452-ac95-a758aec35d52","Type":"ContainerStarted","Data":"7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04"} Oct 29 22:46:12.227225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.227184 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:12.228734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.228679 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" event={"ID":"0e712137-2012-4419-90e9-95236293218a","Type":"ContainerDied","Data":"b9ca9ea4a92d2dbf885a5b92f8d6ee5a0701a0853fd8a07cb1edcd581f9f749f"} Oct 29 22:46:12.228837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.228741 5367 scope.go:122] "RemoveContainer" containerID="9924244fee9753ea90eac4cde6c3be00f9d602dcdfcdc0ab05238205daf564e8" Oct 29 22:46:12.228837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.228684 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8" Oct 29 22:46:12.308967 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.308907 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" podStartSLOduration=3.3088903849999998 podStartE2EDuration="3.308890385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:12.277331402 +0000 UTC m=+1839.171406324" watchObservedRunningTime="2026-10-29 22:46:12.308890385 +0000 UTC m=+1839.202965304" Oct 29 22:46:12.327371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.327336 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8"] Oct 29 22:46:12.332300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.332267 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-63db1a-988f85d4-9x9r8"] Oct 29 22:46:12.405723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.405026 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" podStartSLOduration=3.40500488 podStartE2EDuration="3.40500488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:12.379075813 +0000 UTC m=+1839.273150734" watchObservedRunningTime="2026-10-29 22:46:12.40500488 +0000 UTC m=+1839.299079801" Oct 29 22:46:12.981736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.980842 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.981736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.980914 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 22:46:12.996304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.996269 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.996560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:12.996532 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 29 22:46:13.230643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:13.230598 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e712137-2012-4419-90e9-95236293218a" path="/var/lib/kubelet/pods/0e712137-2012-4419-90e9-95236293218a/volumes" Oct 29 22:46:13.231412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:13.231381 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81bdf6c-42eb-40d3-8d1f-cc2976e61529" path="/var/lib/kubelet/pods/c81bdf6c-42eb-40d3-8d1f-cc2976e61529/volumes" Oct 29 22:46:13.981147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:13.981116 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:46:13.982247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:13.982221 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:46:13.990009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:13.989984 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:46:13.990855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:13.990835 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:46:19.658374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:19.658346 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:46:20.416527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:20.416490 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr"] Oct 29 22:46:20.416809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:20.416781 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" containerID="cri-o://ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97" gracePeriod=30 Oct 29 22:46:20.431210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:20.431127 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85"] Oct 29 22:46:20.431389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:20.431346 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" containerID="cri-o://4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2" gracePeriod=30 Oct 29 22:46:20.476796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:20.476765 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q"] Oct 29 22:46:20.476957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:20.476937 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" containerID="cri-o://a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056" gracePeriod=30 Oct 29 22:46:21.226264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.226235 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:46:21.733723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.733310 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8"] Oct 29 22:46:21.735260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.735229 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0e712137-2012-4419-90e9-95236293218a" containerName="maas-api" Oct 29 22:46:21.735534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.735269 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e712137-2012-4419-90e9-95236293218a" containerName="maas-api" Oct 29 22:46:21.735614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.735593 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e712137-2012-4419-90e9-95236293218a" containerName="maas-api" Oct 29 22:46:21.750482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.741484 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.762719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.761234 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8"] Oct 29 22:46:21.797908 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.797848 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-45e508-openshift-default-dockercfg-lpjww\"" Oct 29 22:46:21.802139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802108 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802277 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802357 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802466 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802553 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802582 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwnsc\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802672 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802750 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.802902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.802887 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908023 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908313 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908395 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908483 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908512 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwnsc\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908628 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908710 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908886 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908963 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.911872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.908974 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.931462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.931419 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.931664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.931633 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.931980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.931933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.932459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.932439 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.938313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.938141 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.944356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.944326 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.946130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.945922 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwnsc\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:21.959111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:21.959081 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:22.066252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:22.066212 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:22.331962 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:22.331872 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:46:22.396745 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:22.396669 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea7308_86cf_448e_ae51_090ff57081ff.slice/crio-cd234faf4b1b39f55295e57907934c3da88226cc9802df6601022c91525f621d WatchSource:0}: Error finding container cd234faf4b1b39f55295e57907934c3da88226cc9802df6601022c91525f621d: Status 404 returned error can't find the container with id cd234faf4b1b39f55295e57907934c3da88226cc9802df6601022c91525f621d Oct 29 22:46:22.396910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:22.396890 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8"] Oct 29 22:46:22.400091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:22.400053 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:22.400182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:22.400139 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:22.400219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:22.400181 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:23.246236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:23.246207 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:46:23.246436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:23.246268 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:46:23.350060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:23.350025 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" event={"ID":"7eea7308-86cf-448e-ae51-090ff57081ff","Type":"ContainerStarted","Data":"85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea"} Oct 29 22:46:23.350532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:23.350082 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" event={"ID":"7eea7308-86cf-448e-ae51-090ff57081ff","Type":"ContainerStarted","Data":"cd234faf4b1b39f55295e57907934c3da88226cc9802df6601022c91525f621d"} Oct 29 22:46:23.406535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:23.406483 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podStartSLOduration=2.406468598 podStartE2EDuration="2.406468598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:23.400855967 +0000 UTC m=+1850.294930889" watchObservedRunningTime="2026-10-29 22:46:23.406468598 +0000 UTC m=+1850.300543516" Oct 29 22:46:24.067037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:24.067002 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:24.073227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:24.073202 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:24.360033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:24.359953 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:24.360823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:24.360803 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:46:24.549564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:24.549521 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:24.549794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:24.549592 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:25.411935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:25.411896 5367 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:25.412937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:25.411972 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:25.844842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:25.844797 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:46:25.916145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:25.916117 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:46:25.988173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:25.988146 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:46:26.009784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.009751 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.010005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.009991 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.010144 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010135 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.010270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010258 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.010388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010380 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.010507 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010496 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.010575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010567 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.010637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010627 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.010749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010739 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.010860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010850 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.010924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010916 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.011021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011010 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.011107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011099 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.011174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011166 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.011270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011262 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.011436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011422 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.011622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011608 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data\") pod \"d1e03932-f445-49d2-85dc-6f89dd2657dc\" (UID: \"d1e03932-f445-49d2-85dc-6f89dd2657dc\") " Oct 29 22:46:26.011778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011763 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs\") pod \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\" (UID: \"5ee79d16-2b7e-4129-8cff-8235bde26d3b\") " Oct 29 22:46:26.014469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010757 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.010953 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.011641 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.013022 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.013218 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.013498 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.013760 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.013798 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.014033 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.014588 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.014262 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.017606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.017577 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.021095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.021066 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "kube-api-access-grqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.021211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.021065 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.021211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.021121 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy" pod "d1e03932-f445-49d2-85dc-6f89dd2657dc" (UID: "d1e03932-f445-49d2-85dc-6f89dd2657dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.021211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.021186 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.021211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.021194 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.021211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.021202 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "kube-api-access-bz27p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.022310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.022280 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy" pod "5ee79d16-2b7e-4129-8cff-8235bde26d3b" (UID: "5ee79d16-2b7e-4129-8cff-8235bde26d3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.114211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114179 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114216 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114324 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114354 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114394 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114471 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114510 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114526 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.114819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114623 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token\") pod \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\" (UID: \"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424\") " Oct 29 22:46:26.114985 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.114837 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.115648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115627 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.115778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115651 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.115778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115664 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.115778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115677 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.115778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115688 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.115778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115723 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.115778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115736 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115770 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115795 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115819 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115833 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115867 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqgn\" (UniqueName: \"kubernetes.io/projected/d1e03932-f445-49d2-85dc-6f89dd2657dc-kube-api-access-grqgn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115880 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz27p\" (UniqueName: \"kubernetes.io/projected/5ee79d16-2b7e-4129-8cff-8235bde26d3b-kube-api-access-bz27p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115894 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115908 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115921 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e03932-f445-49d2-85dc-6f89dd2657dc-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115933 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115944 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.115955 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee79d16-2b7e-4129-8cff-8235bde26d3b-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.116805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.116316 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e03932-f445-49d2-85dc-6f89dd2657dc-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.117012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.116970 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.117114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.117084 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "kube-api-access-fxxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.117188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.117064 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.117188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.117178 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.117296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.117284 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.118336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.118311 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.118820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.118797 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy" pod "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" (UID: "bb4bcb1d-8b5c-4223-aae8-13ad38dfd424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218099 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218129 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxmn\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-kube-api-access-fxxmn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218139 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218147 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218155 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218179 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.218197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.218193 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.383512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.383478 5367 generic.go:356] "Generic (PLEG): container finished" podID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerID="4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2" exitCode=0 Oct 29 22:46:26.383741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.383539 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" Oct 29 22:46:26.383741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.383709 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" event={"ID":"5ee79d16-2b7e-4129-8cff-8235bde26d3b","Type":"ContainerDied","Data":"4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2"} Oct 29 22:46:26.383885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.383803 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85" event={"ID":"5ee79d16-2b7e-4129-8cff-8235bde26d3b","Type":"ContainerDied","Data":"a18b9ecc0cca93928aa52362601e6f8abb7e17486a6b43c889d761e373724f7e"} Oct 29 22:46:26.383885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.383846 5367 scope.go:122] "RemoveContainer" containerID="4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2" Oct 29 22:46:26.388780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.388744 5367 generic.go:356] "Generic (PLEG): container finished" podID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerID="ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97" exitCode=0 Oct 29 22:46:26.388971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.388852 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" event={"ID":"d1e03932-f445-49d2-85dc-6f89dd2657dc","Type":"ContainerDied","Data":"ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97"} Oct 29 22:46:26.388971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.388890 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" event={"ID":"d1e03932-f445-49d2-85dc-6f89dd2657dc","Type":"ContainerDied","Data":"fc9410ca632d458ef930e012d6ce91c0241245d44b936a5834f336aaaf3b61c4"} Oct 29 22:46:26.388971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.388977 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr" Oct 29 22:46:26.396416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.396384 5367 generic.go:356] "Generic (PLEG): container finished" podID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerID="a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056" exitCode=0 Oct 29 22:46:26.396540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.396454 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" Oct 29 22:46:26.396540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.396465 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" event={"ID":"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424","Type":"ContainerDied","Data":"a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056"} Oct 29 22:46:26.396627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.396543 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q" event={"ID":"bb4bcb1d-8b5c-4223-aae8-13ad38dfd424","Type":"ContainerDied","Data":"23a80716d4dc0567958654f4e873f66c98a46dc71f1d56ae13dcc2ec800dcbc3"} Oct 29 22:46:26.401088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.401072 5367 scope.go:122] "RemoveContainer" containerID="51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21" Oct 29 22:46:26.412264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.412244 5367 scope.go:122] "RemoveContainer" containerID="4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2" Oct 29 22:46:26.412590 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:26.412516 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2\": container with ID starting with 4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2 not found: ID does not exist" containerID="4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2" Oct 29 22:46:26.412590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.412548 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2"} err="failed to get container status \"4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2\": rpc error: code = NotFound desc = could not find container \"4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2\": container with ID starting with 4cc13fa2edf941b3be53aab9d7f398124ca541fcb9930376fb495d1bb09127d2 not found: ID does not exist" Oct 29 22:46:26.412590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.412572 5367 scope.go:122] "RemoveContainer" containerID="51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21" Oct 29 22:46:26.412895 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:26.412876 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21\": container with ID starting with 51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21 not found: ID does not exist" containerID="51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21" Oct 29 22:46:26.412960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.412904 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21"} err="failed to get container status \"51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21\": rpc error: code = NotFound desc = could not find container \"51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21\": container with ID starting with 51c51caed3f73b69246b9fee5e1e9a7d904f09fd34c15a3afd4ac26c637fec21 not found: ID does not exist" Oct 29 22:46:26.412960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.412918 5367 scope.go:122] "RemoveContainer" containerID="ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97" Oct 29 22:46:26.426583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.426562 5367 scope.go:122] "RemoveContainer" containerID="ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97" Oct 29 22:46:26.426922 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:26.426898 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97\": container with ID starting with ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97 not found: ID does not exist" containerID="ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97" Oct 29 22:46:26.427002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.426930 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97"} err="failed to get container status \"ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97\": rpc error: code = NotFound desc = could not find container \"ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97\": container with ID starting with ecbb5e7ac507bd95b84d3f393e3633e67d8351dd4c88ef364977cf132aff1f97 not found: ID does not exist" Oct 29 22:46:26.427002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.426954 5367 scope.go:122] "RemoveContainer" containerID="a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056" Oct 29 22:46:26.437623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.437605 5367 scope.go:122] "RemoveContainer" containerID="a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056" Oct 29 22:46:26.437912 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:26.437892 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056\": container with ID starting with a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056 not found: ID does not exist" containerID="a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056" Oct 29 22:46:26.437983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.437922 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056"} err="failed to get container status \"a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056\": rpc error: code = NotFound desc = could not find container \"a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056\": container with ID starting with a0687b41ca626a8b873f60638b48b58824a7905291842762cc495be9b3f1c056 not found: ID does not exist" Oct 29 22:46:26.464478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.464451 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85"] Oct 29 22:46:26.494543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.494468 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-chw85"] Oct 29 22:46:26.591571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.591535 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr"] Oct 29 22:46:26.598851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.598820 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-dlmxr"] Oct 29 22:46:26.636991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.636959 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q"] Oct 29 22:46:26.646003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:26.645974 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-8b69q"] Oct 29 22:46:27.225552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.225519 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" path="/var/lib/kubelet/pods/5ee79d16-2b7e-4129-8cff-8235bde26d3b/volumes" Oct 29 22:46:27.226255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.226238 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" path="/var/lib/kubelet/pods/bb4bcb1d-8b5c-4223-aae8-13ad38dfd424/volumes" Oct 29 22:46:27.226883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.226867 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" path="/var/lib/kubelet/pods/d1e03932-f445-49d2-85dc-6f89dd2657dc/volumes" Oct 29 22:46:27.843527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.843486 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg"] Oct 29 22:46:27.844476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844451 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" Oct 29 22:46:27.844580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844477 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" Oct 29 22:46:27.844580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844514 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" Oct 29 22:46:27.844580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844522 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" Oct 29 22:46:27.844580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844538 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" Oct 29 22:46:27.844580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844545 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" Oct 29 22:46:27.844872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844794 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e03932-f445-49d2-85dc-6f89dd2657dc" containerName="istio-proxy" Oct 29 22:46:27.844872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844819 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4bcb1d-8b5c-4223-aae8-13ad38dfd424" containerName="istio-proxy" Oct 29 22:46:27.844872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.844827 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" Oct 29 22:46:27.849290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.849272 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:27.867330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.867309 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-45e508\"" Oct 29 22:46:27.880367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.880335 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-45e508-dockercfg-2ghbd\"" Oct 29 22:46:27.888450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.888413 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg"] Oct 29 22:46:27.964724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.956373 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:27.964724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:27.956544 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsdh\" (UniqueName: \"kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh\") pod \"payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:28.054727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.001766 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8"] Oct 29 22:46:28.054727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.046842 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" Oct 29 22:46:28.054727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.046863 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" Oct 29 22:46:28.054727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.047043 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee79d16-2b7e-4129-8cff-8235bde26d3b" containerName="istio-proxy" Oct 29 22:46:28.064725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.063142 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:28.064725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.063318 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsdh\" (UniqueName: \"kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh\") pod \"payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:28.064988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.064862 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:28.086719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.084205 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8"] Oct 29 22:46:28.086719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.084326 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.106721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.104572 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsdh\" (UniqueName: \"kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh\") pod \"payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:28.174967 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.174455 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:28.177003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.176972 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpfz\" (UniqueName: \"kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz\") pod \"payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.177340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.177264 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.278595 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.278561 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.278994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.278969 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpfz\" (UniqueName: \"kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz\") pod \"payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.279466 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.279446 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.302676 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.302650 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpfz\" (UniqueName: \"kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz\") pod \"payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.430034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.429954 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:28.492711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.492668 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg"] Oct 29 22:46:28.497345 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:28.497315 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454d48ec_548e_4a85_beba_82560fdfb3aa.slice/crio-4d3b84b29d5dd7c1c691245f3fecbab72af11356afcb39cc5b06cd504bdc9a7f WatchSource:0}: Error finding container 4d3b84b29d5dd7c1c691245f3fecbab72af11356afcb39cc5b06cd504bdc9a7f: Status 404 returned error can't find the container with id 4d3b84b29d5dd7c1c691245f3fecbab72af11356afcb39cc5b06cd504bdc9a7f Oct 29 22:46:28.740779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:28.740750 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8"] Oct 29 22:46:28.741745 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:28.741708 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d01f693_6f67_46ae_a7fd_0e6f7a48e520.slice/crio-36abc9041346f8a052631b6c88c782057c795b733abd599bade6db640abc0e64 WatchSource:0}: Error finding container 36abc9041346f8a052631b6c88c782057c795b733abd599bade6db640abc0e64: Status 404 returned error can't find the container with id 36abc9041346f8a052631b6c88c782057c795b733abd599bade6db640abc0e64 Oct 29 22:46:29.426469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.426426 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" event={"ID":"454d48ec-548e-4a85-beba-82560fdfb3aa","Type":"ContainerStarted","Data":"09f3f20c13ad954973e5818af072fae1d9a395af87759d87569cb4d898a8f8f8"} Oct 29 22:46:29.426956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.426494 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" event={"ID":"454d48ec-548e-4a85-beba-82560fdfb3aa","Type":"ContainerStarted","Data":"4d3b84b29d5dd7c1c691245f3fecbab72af11356afcb39cc5b06cd504bdc9a7f"} Oct 29 22:46:29.426956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.426655 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:29.428266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.428244 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" event={"ID":"1d01f693-6f67-46ae-a7fd-0e6f7a48e520","Type":"ContainerStarted","Data":"63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051"} Oct 29 22:46:29.428403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.428279 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" event={"ID":"1d01f693-6f67-46ae-a7fd-0e6f7a48e520","Type":"ContainerStarted","Data":"36abc9041346f8a052631b6c88c782057c795b733abd599bade6db640abc0e64"} Oct 29 22:46:29.428403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.428314 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:29.471767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.471711 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" podStartSLOduration=2.471685919 podStartE2EDuration="2.471685919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:29.468402204 +0000 UTC m=+1856.362477136" watchObservedRunningTime="2026-10-29 22:46:29.471685919 +0000 UTC m=+1856.365760838" Oct 29 22:46:29.515087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:29.515040 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" podStartSLOduration=2.515025814 podStartE2EDuration="2.515025814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:29.51268211 +0000 UTC m=+1856.406757030" watchObservedRunningTime="2026-10-29 22:46:29.515025814 +0000 UTC m=+1856.409100735" Oct 29 22:46:36.813663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:36.813628 5367 scope.go:122] "RemoveContainer" containerID="b372505febf57337b95d6a74986e9c60faef4ccefa94d8e35a619b6293f13c2b" Oct 29 22:46:36.826726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:36.826687 5367 scope.go:122] "RemoveContainer" containerID="c08c2a3dbfa8a31b84b1e9b894a7696abd13694d47c3e0129e0b2e8383552983" Oct 29 22:46:36.837463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:36.837447 5367 scope.go:122] "RemoveContainer" containerID="b4bb0ca2a093fa4cf5e5b927da4aeafc2bb5636ba5e2ea46e3e24e5f293d05fb" Oct 29 22:46:40.452473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:40.452438 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:46:40.453133 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:40.453111 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:46:47.918241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:47.918200 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv"] Oct 29 22:46:47.932638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:47.932606 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e224cd-898jv"] Oct 29 22:46:49.226407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:49.226365 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff7526d-938c-40ec-8e34-942e9e7a08ae" path="/var/lib/kubelet/pods/1ff7526d-938c-40ec-8e34-942e9e7a08ae/volumes" Oct 29 22:46:50.824401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.824368 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg"] Oct 29 22:46:50.831536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.831514 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.855581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.855545 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg"] Oct 29 22:46:50.865770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.865738 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.865948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.865801 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.865948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.865826 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.865948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.865865 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.865948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.865908 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.865948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.865928 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89bh\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.866228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.866024 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.866228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.866044 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.866228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.866204 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.870964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.870937 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk"] Oct 29 22:46:50.877261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.877241 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.914753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.908135 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk"] Oct 29 22:46:50.963550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.963517 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7"] Oct 29 22:46:50.963769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.963736 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" containerID="cri-o://78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936" gracePeriod=30 Oct 29 22:46:50.968238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968211 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968272 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968307 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968332 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968433 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968482 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968522 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968563 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968597 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89bh\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968626 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968652 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968802 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968832 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968882 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968856 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.968990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968919 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968945 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zkc\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.968990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.968979 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.969102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.969085 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.969141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.969101 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:50.969630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.969607 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.969772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.969753 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.969888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.969871 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.970323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.970274 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.970912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.970886 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.972202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.972181 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.993892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.993861 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:50.995136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:50.995111 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89bh\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:51.075665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.075561 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.075665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.075613 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.075956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.075803 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.075956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.075833 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zkc\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.075956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.075862 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.075961 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076067 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076217 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076275 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076287 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076358 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076368 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076538 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.076610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.076586 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.078521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.078497 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.080361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.079807 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.086336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.086314 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b"] Oct 29 22:46:51.086550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.086529 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" containerID="cri-o://e1e44f355bacbcffadad4311676f349667e15786a427dd8d7e965d3f298938fb" gracePeriod=30 Oct 29 22:46:51.112109 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.112081 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zkc\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.115072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.115041 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.146663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.146629 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:51.213115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.212365 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:51.486866 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:51.486836 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530ad7df_e19e_4555_aab1_cc28df621b62.slice/crio-6994ddeb3d754c9f95304512ed69ad76fe353f77b92eea6222031c1361cac791 WatchSource:0}: Error finding container 6994ddeb3d754c9f95304512ed69ad76fe353f77b92eea6222031c1361cac791: Status 404 returned error can't find the container with id 6994ddeb3d754c9f95304512ed69ad76fe353f77b92eea6222031c1361cac791 Oct 29 22:46:51.488216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.488189 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg"] Oct 29 22:46:51.490035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.489998 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:51.490225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.490195 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:51.490338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.490265 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:51.509976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.509945 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x"] Oct 29 22:46:51.510157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.510137 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" containerID="cri-o://89896b35ebc64bbe7f78e715ce6665e848e6e03631b82cbb749e1811ad4eb8e4" gracePeriod=30 Oct 29 22:46:51.562493 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:51.562445 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec53c939_317f_4bab_a39e_0c4e722e5050.slice/crio-f2e5874ff8bfcd7473255d754e0d9ac6eabe57e8064bfe4bc43c122f09115ab7 WatchSource:0}: Error finding container f2e5874ff8bfcd7473255d754e0d9ac6eabe57e8064bfe4bc43c122f09115ab7: Status 404 returned error can't find the container with id f2e5874ff8bfcd7473255d754e0d9ac6eabe57e8064bfe4bc43c122f09115ab7 Oct 29 22:46:51.567183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.567143 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:51.567301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.567230 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:51.567301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.567271 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:46:51.567843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.567820 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk"] Oct 29 22:46:51.770153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.770053 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" event={"ID":"530ad7df-e19e-4555-aab1-cc28df621b62","Type":"ContainerStarted","Data":"410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a"} Oct 29 22:46:51.770153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.770117 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" event={"ID":"530ad7df-e19e-4555-aab1-cc28df621b62","Type":"ContainerStarted","Data":"6994ddeb3d754c9f95304512ed69ad76fe353f77b92eea6222031c1361cac791"} Oct 29 22:46:51.772215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.772186 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" event={"ID":"ec53c939-317f-4bab-a39e-0c4e722e5050","Type":"ContainerStarted","Data":"028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa"} Oct 29 22:46:51.772347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.772239 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" event={"ID":"ec53c939-317f-4bab-a39e-0c4e722e5050","Type":"ContainerStarted","Data":"f2e5874ff8bfcd7473255d754e0d9ac6eabe57e8064bfe4bc43c122f09115ab7"} Oct 29 22:46:51.815157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.815107 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podStartSLOduration=1.8150936629999999 podStartE2EDuration="1.815093663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:51.813985884 +0000 UTC m=+1878.708060805" watchObservedRunningTime="2026-10-29 22:46:51.815093663 +0000 UTC m=+1878.709168584" Oct 29 22:46:51.862748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:51.862677 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podStartSLOduration=1.8626626769999999 podStartE2EDuration="1.862662677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:51.859479543 +0000 UTC m=+1878.753554477" watchObservedRunningTime="2026-10-29 22:46:51.862662677 +0000 UTC m=+1878.756737597" Oct 29 22:46:52.147871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:52.147837 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:52.149812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:52.149784 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:52.149983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:52.149839 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:52.214257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:52.214228 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:52.215875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:52.215847 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:52.216016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:52.215901 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:52.332435 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:52.332401 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:46:53.147587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:53.147553 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:53.148129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:53.148103 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:53.224203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:53.224166 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:53.224484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:53.224456 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:54.147132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:54.147098 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:54.147323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:54.147156 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:54.216470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:54.216438 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:54.216947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:54.216489 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:54.989720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:54.988913 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6"] Oct 29 22:46:55.164201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.158848 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:55.164201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.158957 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:55.273774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.269512 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:55.273774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.269582 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:55.288117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.283453 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6"] Oct 29 22:46:55.288117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.284749 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.337538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.337494 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8dd4c8-serving-cert\"" Oct 29 22:46:55.412229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.412193 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.412418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.412307 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.412569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.412557 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.412656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.412644 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpmb\" (UniqueName: \"kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.514522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.514490 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.514727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.514573 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpmb\" (UniqueName: \"kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.514727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.514607 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.514727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.514669 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.515157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.515131 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.518375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.518343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.518545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.518524 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.534581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.534494 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpmb\" (UniqueName: \"kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb\") pod \"maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:55.673274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:55.673150 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:56.017267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.017217 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:56.017496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.017442 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:56.048527 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:46:56.048497 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb084b8e7_3416_4e01_a52f_3dbb55a6b0ec.slice/crio-f7d5a7168ecadbf02d1564ff0537ec146e73db1f06d52a637b64065bfa23b719 WatchSource:0}: Error finding container f7d5a7168ecadbf02d1564ff0537ec146e73db1f06d52a637b64065bfa23b719: Status 404 returned error can't find the container with id f7d5a7168ecadbf02d1564ff0537ec146e73db1f06d52a637b64065bfa23b719 Oct 29 22:46:56.049087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.049056 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6"] Oct 29 22:46:56.114788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.114010 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:46:56.149523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.149104 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:56.149523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.149155 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:56.225721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.224373 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:56.225721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.224433 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:56.441014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.440987 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:46:56.553097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.552486 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.552579 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.552619 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.552669 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.552931 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.552976 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.553101 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.553300 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.553358 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket\") pod \"746b797f-764a-4e4e-ab9a-8c61dea134ee\" (UID: \"746b797f-764a-4e4e-ab9a-8c61dea134ee\") " Oct 29 22:46:56.553645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.553576 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.568722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.554168 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.590724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.570878 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.590724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.589734 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.590724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.590079 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:56.642460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.607830 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.682724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.676064 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.682724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.676102 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.682724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.676116 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746b797f-764a-4e4e-ab9a-8c61dea134ee-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.682724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.676129 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.801968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.798739 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.801968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.801915 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.828876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.828515 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:56.830296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.830266 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "kube-api-access-gbhgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:56.830892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.830850 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token" pod "746b797f-764a-4e4e-ab9a-8c61dea134ee" (UID: "746b797f-764a-4e4e-ab9a-8c61dea134ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:56.903925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.903431 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.903925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.903473 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-kube-api-access-gbhgf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.903925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:56.903489 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746b797f-764a-4e4e-ab9a-8c61dea134ee-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.046535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.046500 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" event={"ID":"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec","Type":"ContainerStarted","Data":"e43fb67458754f7e8093c23860be8994867ff382f0f8824d750fe26a7af5c1b3"} Oct 29 22:46:57.046795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.046769 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" event={"ID":"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec","Type":"ContainerStarted","Data":"f7d5a7168ecadbf02d1564ff0537ec146e73db1f06d52a637b64065bfa23b719"} Oct 29 22:46:57.046946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.046933 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:46:57.049741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.049420 5367 generic.go:356] "Generic (PLEG): container finished" podID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerID="89896b35ebc64bbe7f78e715ce6665e848e6e03631b82cbb749e1811ad4eb8e4" exitCode=0 Oct 29 22:46:57.049741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.049611 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" event={"ID":"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0","Type":"ContainerDied","Data":"89896b35ebc64bbe7f78e715ce6665e848e6e03631b82cbb749e1811ad4eb8e4"} Oct 29 22:46:57.052643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.052618 5367 generic.go:356] "Generic (PLEG): container finished" podID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerID="78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936" exitCode=0 Oct 29 22:46:57.053017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.052866 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" Oct 29 22:46:57.053384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.052878 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" event={"ID":"746b797f-764a-4e4e-ab9a-8c61dea134ee","Type":"ContainerDied","Data":"78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936"} Oct 29 22:46:57.053384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.053229 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7" event={"ID":"746b797f-764a-4e4e-ab9a-8c61dea134ee","Type":"ContainerDied","Data":"b48d7dad9e532029c3e1130afbb07bd1d31262e6d926dc43cbb799c4059c7795"} Oct 29 22:46:57.053384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.053269 5367 scope.go:122] "RemoveContainer" containerID="78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936" Oct 29 22:46:57.118757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.118634 5367 generic.go:356] "Generic (PLEG): container finished" podID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerID="e1e44f355bacbcffadad4311676f349667e15786a427dd8d7e965d3f298938fb" exitCode=0 Oct 29 22:46:57.119121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.119054 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" event={"ID":"5f1ff5d8-5462-4302-9296-fdf9a4597911","Type":"ContainerDied","Data":"e1e44f355bacbcffadad4311676f349667e15786a427dd8d7e965d3f298938fb"} Oct 29 22:46:57.131682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.131661 5367 scope.go:122] "RemoveContainer" containerID="78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936" Oct 29 22:46:57.132178 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:46:57.132146 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936\": container with ID starting with 78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936 not found: ID does not exist" containerID="78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936" Oct 29 22:46:57.132261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.132191 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936"} err="failed to get container status \"78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936\": rpc error: code = NotFound desc = could not find container \"78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936\": container with ID starting with 78a36d978054da5650df1d7c5c141ffb9f3a0e69f01b8270be1ccef34f68f936 not found: ID does not exist" Oct 29 22:46:57.142658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.142605 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" podStartSLOduration=2.80549164 podStartE2EDuration="3.142590766s" totalImagesPullingTime="337.099126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:57.13976675 +0000 UTC m=+1884.033841670" watchObservedRunningTime="2026-10-29 22:46:57.142590766 +0000 UTC m=+1884.036665680" Oct 29 22:46:57.147114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.147091 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:57.147241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.147145 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:57.224403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.219923 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:57.224403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.219986 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:57.246536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.246454 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:46:57.249957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.249931 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7"] Oct 29 22:46:57.251647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.250753 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-wh4d7"] Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.431441 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.443871 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.443948 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.444065 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.444132 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.444167 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.444221 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.444448 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.444503 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s\") pod \"5f1ff5d8-5462-4302-9296-fdf9a4597911\" (UID: \"5f1ff5d8-5462-4302-9296-fdf9a4597911\") " Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.439978 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.449156 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.432345 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.449735 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.449791 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.457734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.449836 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.533732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.533681 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "kube-api-access-vhx9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.535240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.535211 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.535364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.535214 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:57.535364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.535244 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy" pod "5f1ff5d8-5462-4302-9296-fdf9a4597911" (UID: "5f1ff5d8-5462-4302-9296-fdf9a4597911"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.548062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548030 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548083 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548120 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548234 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548297 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548417 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548496 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548550 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.548674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.548640 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56\") pod \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\" (UID: \"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0\") " Oct 29 22:46:57.549369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549347 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549377 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhx9s\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-kube-api-access-vhx9s\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549393 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ff5d8-5462-4302-9296-fdf9a4597911-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549406 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549418 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549430 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549443 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549455 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.549677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.549466 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ff5d8-5462-4302-9296-fdf9a4597911-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.552458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.552228 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "kube-api-access-g5w56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.552923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.552619 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.553215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.553193 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.561623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.556884 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:57.605136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.595005 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.606914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.606849 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.628451 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.628385 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:57.629921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.629867 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.636831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.636793 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy" pod "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" (UID: "6d26976f-bf8d-40aa-8d2b-3fe83a092ed0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651718 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651867 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651880 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651924 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5w56\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-kube-api-access-g5w56\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651938 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651950 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651963 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651983 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.675982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:57.651996 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:46:58.006846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.006757 5367 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:46:58.007102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.007075 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:46:58.135152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.135123 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" Oct 29 22:46:58.135360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.135159 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b" event={"ID":"5f1ff5d8-5462-4302-9296-fdf9a4597911","Type":"ContainerDied","Data":"f3dc9c0ec940ad77786d347575e6b0008c6777294ec2c1f1a55134b96ed703d1"} Oct 29 22:46:58.135360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.135231 5367 scope.go:122] "RemoveContainer" containerID="e1e44f355bacbcffadad4311676f349667e15786a427dd8d7e965d3f298938fb" Oct 29 22:46:58.136999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.136956 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" Oct 29 22:46:58.136999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.136975 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x" event={"ID":"6d26976f-bf8d-40aa-8d2b-3fe83a092ed0","Type":"ContainerDied","Data":"0d1994888b502c69ad1c6cd6da924cb1308794800a7f113e0563e92875cb4b85"} Oct 29 22:46:58.147287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.147041 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 22:46:58.147287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.147089 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 22:46:58.149062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.149037 5367 scope.go:122] "RemoveContainer" containerID="89896b35ebc64bbe7f78e715ce6665e848e6e03631b82cbb749e1811ad4eb8e4" Oct 29 22:46:58.195618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.195580 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x"] Oct 29 22:46:58.202295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.202262 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-f2m6x"] Oct 29 22:46:58.213875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.213842 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:46:58.214036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.213901 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 29 22:46:58.238807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.238770 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b"] Oct 29 22:46:58.242901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:58.242875 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nvx8b"] Oct 29 22:46:59.155130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.155095 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:59.155740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.155651 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:59.157103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.157083 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:46:59.228416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.228370 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" path="/var/lib/kubelet/pods/5f1ff5d8-5462-4302-9296-fdf9a4597911/volumes" Oct 29 22:46:59.229001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.228984 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" path="/var/lib/kubelet/pods/6d26976f-bf8d-40aa-8d2b-3fe83a092ed0/volumes" Oct 29 22:46:59.229441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.229428 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" path="/var/lib/kubelet/pods/746b797f-764a-4e4e-ab9a-8c61dea134ee/volumes" Oct 29 22:46:59.230248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.230228 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:59.230776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.230751 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:46:59.231400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:46:59.231381 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:47:03.143189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:03.143160 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:47:08.489511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.489474 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7"] Oct 29 22:47:08.490270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490256 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" Oct 29 22:47:08.490316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490271 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" Oct 29 22:47:08.490316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490285 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" Oct 29 22:47:08.490316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490290 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" Oct 29 22:47:08.490437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490329 5367 cpu_manager.go:414] "containerMap: removing container" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" Oct 29 22:47:08.490437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490335 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" Oct 29 22:47:08.490510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490474 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="746b797f-764a-4e4e-ab9a-8c61dea134ee" containerName="istio-proxy" Oct 29 22:47:08.490510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490487 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1ff5d8-5462-4302-9296-fdf9a4597911" containerName="istio-proxy" Oct 29 22:47:08.490510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.490496 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d26976f-bf8d-40aa-8d2b-3fe83a092ed0" containerName="istio-proxy" Oct 29 22:47:08.496312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.496295 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.514808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.514775 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7"] Oct 29 22:47:08.522380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.522309 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-45e508-serving-cert\"" Oct 29 22:47:08.582810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.582772 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.582979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.582833 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qgf\" (UniqueName: \"kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.582979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.582901 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.582979 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.582946 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.698719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.687383 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.705815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.701305 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qgf\" (UniqueName: \"kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.705815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.701365 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.705815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.701425 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.705815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.702365 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.705815 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:47:08.700261 5367 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-serving-cert: secret "maas-api-e2e-inf-a-45e508-serving-cert" not found Oct 29 22:47:08.705815 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:47:08.702520 5367 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls podName:575769df-b144-4255-8d0c-bb402c343d43 nodeName:}" failed. No retries permitted until 2026-10-29 22:47:09.202493087 +0000 UTC m=+1896.096568007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls") pod "maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" (UID: "575769df-b144-4255-8d0c-bb402c343d43") : secret "maas-api-e2e-inf-a-45e508-serving-cert" not found Oct 29 22:47:08.707948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.707918 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:08.726212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:08.726183 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qgf\" (UniqueName: \"kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:09.208949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:09.208913 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:09.211368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:09.211343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:09.411198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:09.411161 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:09.740948 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:47:09.740881 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575769df_b144_4255_8d0c_bb402c343d43.slice/crio-c154a6a4bb70423a937270c8d91266e262dd99a3be077b790305eceebfb65489 WatchSource:0}: Error finding container c154a6a4bb70423a937270c8d91266e262dd99a3be077b790305eceebfb65489: Status 404 returned error can't find the container with id c154a6a4bb70423a937270c8d91266e262dd99a3be077b790305eceebfb65489 Oct 29 22:47:09.741264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:09.741022 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7"] Oct 29 22:47:10.278190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:10.278143 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" event={"ID":"575769df-b144-4255-8d0c-bb402c343d43","Type":"ContainerStarted","Data":"e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f"} Oct 29 22:47:10.278385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:10.278265 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" event={"ID":"575769df-b144-4255-8d0c-bb402c343d43","Type":"ContainerStarted","Data":"c154a6a4bb70423a937270c8d91266e262dd99a3be077b790305eceebfb65489"} Oct 29 22:47:10.279078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:10.279046 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:10.321015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:10.320967 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" podStartSLOduration=1.985938867 podStartE2EDuration="2.320954049s" totalImagesPullingTime="335.015182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:10.319774829 +0000 UTC m=+1897.213849748" watchObservedRunningTime="2026-10-29 22:47:10.320954049 +0000 UTC m=+1897.215028968" Oct 29 22:47:16.292147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:16.292063 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:47:22.333224 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:47:22.333182 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:47:27.216896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:27.216842 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:47:27.542780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:27.532017 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:47:28.835461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:28.835422 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2"] Oct 29 22:47:28.841542 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:28.841520 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:28.893042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:28.893011 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-dbb507\"" Oct 29 22:47:28.901782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:28.893390 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-dbb507-dockercfg-vwcft\"" Oct 29 22:47:29.054727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.033540 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8dlb\" (UniqueName: \"kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb\") pod \"payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:29.054727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.033661 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:29.155793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.155671 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8dlb\" (UniqueName: \"kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb\") pod \"payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:29.156146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.156120 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:29.157509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.157473 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:29.354363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.354322 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2"] Oct 29 22:47:29.366517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.366403 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8dlb\" (UniqueName: \"kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb\") pod \"payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:29.523325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:29.522102 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:30.212207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:30.209849 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2"] Oct 29 22:47:30.792722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:30.787651 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" event={"ID":"a1583b34-c0be-45ae-b6cf-b53ba9f40258","Type":"ContainerStarted","Data":"22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2"} Oct 29 22:47:30.792722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:30.787738 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" event={"ID":"a1583b34-c0be-45ae-b6cf-b53ba9f40258","Type":"ContainerStarted","Data":"0c68ea42c10b2398e49974a5c600f335df49e35774c3bf969d1887b82d27f5d8"} Oct 29 22:47:30.793325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:30.793234 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:30.835314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:30.835250 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" podStartSLOduration=2.835229803 podStartE2EDuration="2.835229803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:30.834400094 +0000 UTC m=+1917.728475042" watchObservedRunningTime="2026-10-29 22:47:30.835229803 +0000 UTC m=+1917.729304727" Oct 29 22:47:34.495945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:34.495844 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:47:34.495945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:34.495935 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:47:36.518850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:36.518812 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:47:36.519252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:36.518877 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:47:36.916358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:36.916327 5367 scope.go:122] "RemoveContainer" containerID="70ca8fa742bbff30d0ef6d5ba38dd8dd19384a4dc07408369ef2afe6cadcb98f" Oct 29 22:47:36.927563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:36.927543 5367 scope.go:122] "RemoveContainer" containerID="0cb4c997496adfd4ddfea5af874426db50dbacfe236efbaebb96588a2fef313e" Oct 29 22:47:37.893358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:37.893330 5367 generic.go:356] "Generic (PLEG): container finished" podID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerID="201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99" exitCode=0 Oct 29 22:47:37.893934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:37.893404 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" event={"ID":"e08200b1-1dcf-440c-b1ef-6a66ba84e01a","Type":"ContainerDied","Data":"201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99"} Oct 29 22:47:37.894013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:37.893972 5367 scope.go:122] "RemoveContainer" containerID="201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99" Oct 29 22:47:37.895805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:37.895768 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:37.895925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:37.895857 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:37.895925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:37.895915 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:38.904264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.904222 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" event={"ID":"e08200b1-1dcf-440c-b1ef-6a66ba84e01a","Type":"ContainerStarted","Data":"83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62"} Oct 29 22:47:38.906141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.906114 5367 generic.go:356] "Generic (PLEG): container finished" podID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerID="e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2" exitCode=0 Oct 29 22:47:38.906269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.906183 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" event={"ID":"c2441bc5-e772-4c00-aca4-5a6dee3314e7","Type":"ContainerDied","Data":"e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2"} Oct 29 22:47:38.906610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.906596 5367 scope.go:122] "RemoveContainer" containerID="e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2" Oct 29 22:47:38.908368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.908342 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:38.908443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.908410 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:38.908484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:38.908453 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:39.578723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:39.578021 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:47:39.578723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:39.578141 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:47:39.578723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:39.578185 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:47:39.917408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:39.917329 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" event={"ID":"c2441bc5-e772-4c00-aca4-5a6dee3314e7","Type":"ContainerStarted","Data":"a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f"} Oct 29 22:47:40.193157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:40.193078 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:47:40.206987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:40.206943 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:47:40.207182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:40.207024 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 29 22:47:40.581721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:40.576222 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:47:40.581721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:40.576310 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:47:41.187190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:41.187155 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:47:41.187821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:41.187790 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 29 22:47:41.573835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:41.573794 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:47:41.574118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:41.574090 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:47:42.186347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:42.186302 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:47:42.189815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:42.189776 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 29 22:47:42.569478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:42.569442 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:47:42.569770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:42.569742 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:47:42.826154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:42.826076 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 22:47:43.201175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:43.201085 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:47:43.201777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:43.201745 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 29 22:47:43.584007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:43.579997 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:47:43.584007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:43.580048 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:47:44.185815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:44.185772 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:47:44.186041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:44.185830 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 29 22:47:44.604604 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:44.604576 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:47:44.605175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:44.605151 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:47:44.606239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:44.606215 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:47:45.185151 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:45.185117 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:47:45.185337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:45.185166 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 29 22:47:46.189322 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.189292 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:47:46.189730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.189630 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:47:46.190398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.190379 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:47:46.772398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.772361 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb"] Oct 29 22:47:46.776818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.776791 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:46.800160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.800131 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ad69b4\"/\"kube-root-ca.crt\"" Oct 29 22:47:46.800365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.800186 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ad69b4\"/\"openshift-service-ca.crt\"" Oct 29 22:47:46.800462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.800298 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ad69b4\"/\"default-dockercfg-2s4cm\"" Oct 29 22:47:46.801007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.800977 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ad69b4\"/\"test-model-ad69b4-kserve-self-signed-certs\"" Oct 29 22:47:46.809651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.809625 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb"] Oct 29 22:47:46.910776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.910660 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:46.910964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.910936 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:46.911029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.911012 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:46.911263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.911242 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6gw\" (UniqueName: \"kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:46.911341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.911308 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:46.911410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:46.911389 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013133 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6gw\" (UniqueName: \"kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013186 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013294 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013384 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013524 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013581 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013523 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013601 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.013843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.013816 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.015828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.015804 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.016061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.016041 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.034130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.034074 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6gw\" (UniqueName: \"kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw\") pod \"test-model-ad69b4-kserve-8544764ff5-vrfgb\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.099722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.097011 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:47.471372 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:47:47.471340 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b64379c_50a1_4938_ac56_5a49ab326919.slice/crio-b0b607ad84915aba7e20cf954fcd0d2d6c3ef272b2fe77486f51a0030de18044 WatchSource:0}: Error finding container b0b607ad84915aba7e20cf954fcd0d2d6c3ef272b2fe77486f51a0030de18044: Status 404 returned error can't find the container with id b0b607ad84915aba7e20cf954fcd0d2d6c3ef272b2fe77486f51a0030de18044 Oct 29 22:47:47.471739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:47.471467 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb"] Oct 29 22:47:48.272516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:48.272479 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" event={"ID":"3b64379c-50a1-4938-ac56-5a49ab326919","Type":"ContainerStarted","Data":"eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a"} Oct 29 22:47:48.272677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:48.272525 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" event={"ID":"3b64379c-50a1-4938-ac56-5a49ab326919","Type":"ContainerStarted","Data":"b0b607ad84915aba7e20cf954fcd0d2d6c3ef272b2fe77486f51a0030de18044"} Oct 29 22:47:48.322777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:48.322719 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" podStartSLOduration=2.322690687 podStartE2EDuration="2.322690687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:48.318471371 +0000 UTC m=+1935.212546294" watchObservedRunningTime="2026-10-29 22:47:48.322690687 +0000 UTC m=+1935.216765608" Oct 29 22:47:49.099913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:49.099877 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:52.332063 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:47:52.332028 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:47:53.484446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.484412 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7"] Oct 29 22:47:53.488672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.488656 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.528067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.528039 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:47:53.528067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.528060 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:47:53.534351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.534314 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7"] Oct 29 22:47:53.535371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.535338 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.536554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.535430 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.536554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.535630 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6lq\" (UniqueName: \"kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.637534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.637500 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6lq\" (UniqueName: \"kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.637721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.637677 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.637768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.637741 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.638355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.638335 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.640131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.640109 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.661130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.661108 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6lq\" (UniqueName: \"kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq\") pod \"maas-api-revoke-sub-deb7eca1-dqhc7\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.803365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.803295 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:53.817465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.817438 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd"] Oct 29 22:47:53.822034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.822009 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.839182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.839156 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-139eb7-openshift-default-dockercfg-zq8wj\"" Oct 29 22:47:53.841123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841084 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841212 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841328 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841358 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841381 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841458 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841599 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841751 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.841838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.841790 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86xq\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.863628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.863599 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd"] Oct 29 22:47:53.944711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944264 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.944711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944366 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.944711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944411 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86xq\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.944711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944519 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.944711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944602 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.945141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944745 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.945141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944776 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.945141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944801 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.945141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.944833 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.946740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.945427 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.946740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.945644 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.946740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.945876 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.946740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.946456 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.946740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.946677 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.947939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.947915 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.948285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.948262 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.967533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.967499 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:53.968278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:53.968208 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86xq\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:54.122825 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:47:54.122793 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2ffabb_8728_449b_8661_d3d18a0f50b5.slice/crio-8981de92cae0d4a96413f34c2d648fc510f5d08ac33d2e2189d7191809316d01 WatchSource:0}: Error finding container 8981de92cae0d4a96413f34c2d648fc510f5d08ac33d2e2189d7191809316d01: Status 404 returned error can't find the container with id 8981de92cae0d4a96413f34c2d648fc510f5d08ac33d2e2189d7191809316d01 Oct 29 22:47:54.123521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.123467 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7"] Oct 29 22:47:54.164626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.164607 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:54.324651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.324617 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" event={"ID":"ec2ffabb-8728-449b-8661-d3d18a0f50b5","Type":"ContainerStarted","Data":"8981de92cae0d4a96413f34c2d648fc510f5d08ac33d2e2189d7191809316d01"} Oct 29 22:47:54.486822 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:47:54.486792 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3624b108_b174_4506_9d31_76991ddb24d8.slice/crio-3369846aae8260a72ad18e14f12a822b2e641c7d636202a048959f5472bc7e0e WatchSource:0}: Error finding container 3369846aae8260a72ad18e14f12a822b2e641c7d636202a048959f5472bc7e0e: Status 404 returned error can't find the container with id 3369846aae8260a72ad18e14f12a822b2e641c7d636202a048959f5472bc7e0e Oct 29 22:47:54.487237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.486978 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd"] Oct 29 22:47:54.489947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.489919 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:54.490032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.489984 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:54.490032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:54.490013 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:47:55.334919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:55.334885 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" event={"ID":"3624b108-b174-4506-9d31-76991ddb24d8","Type":"ContainerStarted","Data":"a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7"} Oct 29 22:47:55.335068 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:55.334937 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" event={"ID":"3624b108-b174-4506-9d31-76991ddb24d8","Type":"ContainerStarted","Data":"3369846aae8260a72ad18e14f12a822b2e641c7d636202a048959f5472bc7e0e"} Oct 29 22:47:55.336654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:55.336627 5367 generic.go:356] "Generic (PLEG): container finished" podID="ec2ffabb-8728-449b-8661-d3d18a0f50b5" containerID="37f16fd7427d84bce9f41dbd13250c33143756c70a5c87e334aed96aed10a77b" exitCode=0 Oct 29 22:47:55.336805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:55.336714 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" event={"ID":"ec2ffabb-8728-449b-8661-d3d18a0f50b5","Type":"ContainerDied","Data":"37f16fd7427d84bce9f41dbd13250c33143756c70a5c87e334aed96aed10a77b"} Oct 29 22:47:55.402186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:55.402133 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podStartSLOduration=2.402119262 podStartE2EDuration="2.402119262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:55.395512346 +0000 UTC m=+1942.289587267" watchObservedRunningTime="2026-10-29 22:47:55.402119262 +0000 UTC m=+1942.296194242" Oct 29 22:47:56.179202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.179161 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:47:56.181878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.181820 5367 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:47:56.182071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.181908 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 22:47:56.628378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.628348 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:56.716400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.716307 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca\") pod \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " Oct 29 22:47:56.716585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.716417 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca\") pod \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " Oct 29 22:47:56.716585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.716538 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq\" (UniqueName: \"kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq\") pod \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\" (UID: \"ec2ffabb-8728-449b-8661-d3d18a0f50b5\") " Oct 29 22:47:56.716834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.716805 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca" pod "ec2ffabb-8728-449b-8661-d3d18a0f50b5" (UID: "ec2ffabb-8728-449b-8661-d3d18a0f50b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:56.717456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.717428 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:47:56.718904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.718855 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq" pod "ec2ffabb-8728-449b-8661-d3d18a0f50b5" (UID: "ec2ffabb-8728-449b-8661-d3d18a0f50b5"). InnerVolumeSpecName "kube-api-access-dp6lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:56.719898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.719874 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca" pod "ec2ffabb-8728-449b-8661-d3d18a0f50b5" (UID: "ec2ffabb-8728-449b-8661-d3d18a0f50b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:56.819492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.819462 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ffabb-8728-449b-8661-d3d18a0f50b5-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:47:56.819492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:56.819491 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp6lq\" (UniqueName: \"kubernetes.io/projected/ec2ffabb-8728-449b-8661-d3d18a0f50b5-kube-api-access-dp6lq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:47:57.100028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.099995 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:57.112776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.112751 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:57.177725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.175022 5367 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:47:57.177725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.175081 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 22:47:57.380909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.380826 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" Oct 29 22:47:57.380909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.380881 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7" event={"ID":"ec2ffabb-8728-449b-8661-d3d18a0f50b5","Type":"ContainerDied","Data":"8981de92cae0d4a96413f34c2d648fc510f5d08ac33d2e2189d7191809316d01"} Oct 29 22:47:57.381336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.380938 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8981de92cae0d4a96413f34c2d648fc510f5d08ac33d2e2189d7191809316d01" Oct 29 22:47:57.392931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:57.392903 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:47:58.170723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:58.170066 5367 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:47:58.170723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:58.170131 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 22:47:59.181724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:59.180741 5367 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:47:59.181724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:47:59.180811 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 22:48:00.172207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:00.172162 5367 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:48:00.172478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:00.172449 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 22:48:01.170875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:01.170831 5367 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:48:01.172557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:01.172508 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 22:48:02.182723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:02.182072 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:48:02.183253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:02.182977 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:48:02.183881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:02.183846 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:48:04.078728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.073042 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l"] Oct 29 22:48:04.078728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.074450 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ec2ffabb-8728-449b-8661-d3d18a0f50b5" containerName="revoke-keys" Oct 29 22:48:04.078728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.074472 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2ffabb-8728-449b-8661-d3d18a0f50b5" containerName="revoke-keys" Oct 29 22:48:04.078728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.074656 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2ffabb-8728-449b-8661-d3d18a0f50b5" containerName="revoke-keys" Oct 29 22:48:04.085636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.085604 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.109549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.109514 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l"] Oct 29 22:48:04.112778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.112747 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-57d336-openshift-default-dockercfg-5kxjc\"" Oct 29 22:48:04.223215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223180 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223231 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223254 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223467 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223578 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223661 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69gp\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223709 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223768 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.223890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.223875 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.331342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.331249 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.351941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.351904 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.352271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.352249 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69gp\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.352910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.352885 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.353082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.353067 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.353334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.353317 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.353500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.353485 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.365914 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.365969 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.357602 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.358224 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.357048 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.332034 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.366901 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.379721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.377002 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.416718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.403605 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.429797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.422551 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69gp\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.430044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.427234 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:04.710156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:04.710065 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:05.082713 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:48:05.082661 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7d7e47_2202_4ea4_adc7_2804d703c55e.slice/crio-6a4a943555f86acced8723b7d96d522449b12a35a0b68976601ad42fcd2e9d35 WatchSource:0}: Error finding container 6a4a943555f86acced8723b7d96d522449b12a35a0b68976601ad42fcd2e9d35: Status 404 returned error can't find the container with id 6a4a943555f86acced8723b7d96d522449b12a35a0b68976601ad42fcd2e9d35 Oct 29 22:48:05.083685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.083654 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l"] Oct 29 22:48:05.085922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.085886 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:48:05.086022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.085957 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:48:05.086022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.085987 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:48:05.598018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.597984 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" event={"ID":"0e7d7e47-2202-4ea4-adc7-2804d703c55e","Type":"ContainerStarted","Data":"fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d"} Oct 29 22:48:05.598228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.598037 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" event={"ID":"0e7d7e47-2202-4ea4-adc7-2804d703c55e","Type":"ContainerStarted","Data":"6a4a943555f86acced8723b7d96d522449b12a35a0b68976601ad42fcd2e9d35"} Oct 29 22:48:05.652158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.652097 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podStartSLOduration=1.652076039 podStartE2EDuration="1.652076039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:05.650011748 +0000 UTC m=+1952.544086674" watchObservedRunningTime="2026-10-29 22:48:05.652076039 +0000 UTC m=+1952.546150962" Oct 29 22:48:05.710252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:05.710214 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:06.719884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:06.719854 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:07.618049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:07.618011 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:07.619086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:07.619062 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:48:09.786915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:09.785436 5367 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": read tcp 10.133.0.2:47580->10.133.0.238:15021: read: connection reset by peer" start-of-body= Oct 29 22:48:09.786915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:09.785494 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": read tcp 10.133.0.2:47580->10.133.0.238:15021: read: connection reset by peer" Oct 29 22:48:10.649736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:10.649665 5367 generic.go:356] "Generic (PLEG): container finished" podID="7eea7308-86cf-448e-ae51-090ff57081ff" containerID="85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea" exitCode=0 Oct 29 22:48:10.649944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:10.649742 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" event={"ID":"7eea7308-86cf-448e-ae51-090ff57081ff","Type":"ContainerDied","Data":"85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea"} Oct 29 22:48:10.650288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:10.650269 5367 scope.go:122] "RemoveContainer" containerID="85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea" Oct 29 22:48:10.652671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:10.652634 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:48:10.652795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:10.652726 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:48:10.652795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:10.652768 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:48:11.685785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:11.685748 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" event={"ID":"7eea7308-86cf-448e-ae51-090ff57081ff","Type":"ContainerStarted","Data":"ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84"} Oct 29 22:48:12.077801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.070061 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:48:12.077801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.070189 5367 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 29 22:48:12.077801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.070243 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 29 22:48:12.400374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.400339 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm"] Oct 29 22:48:12.404400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.404385 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.435270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.434835 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm"] Oct 29 22:48:12.450732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.440107 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:12.450732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.440148 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:48:12.486920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.486850 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.487126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.487044 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.487422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.487401 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrqm\" (UniqueName: \"kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.589469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.589434 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.589664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.589500 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.589885 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.589859 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrqm\" (UniqueName: \"kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.590319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.590299 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.591982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.591961 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.610952 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.610893 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrqm\" (UniqueName: \"kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm\") pod \"maas-api-revoke-sub-a6636e3f-z2wdm\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:12.718816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:12.718787 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:13.044886 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:48:13.044853 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1786c820_4fda_4b71_9490_d207be700219.slice/crio-a0059c9b88308f1d312b5a856066fcb3b810aa8826da0841385913646cfa29a4 WatchSource:0}: Error finding container a0059c9b88308f1d312b5a856066fcb3b810aa8826da0841385913646cfa29a4: Status 404 returned error can't find the container with id a0059c9b88308f1d312b5a856066fcb3b810aa8826da0841385913646cfa29a4 Oct 29 22:48:13.045518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:13.045495 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm"] Oct 29 22:48:13.069493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:13.069043 5367 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 29 22:48:13.069493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:13.069114 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 29 22:48:13.705482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:13.705438 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" event={"ID":"1786c820-4fda-4b71-9490-d207be700219","Type":"ContainerStarted","Data":"ead33f1907994949fcdef6c1b3e719824fd77f44171b1416208497fe5386ea8f"} Oct 29 22:48:13.705690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:13.705503 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" event={"ID":"1786c820-4fda-4b71-9490-d207be700219","Type":"ContainerStarted","Data":"a0059c9b88308f1d312b5a856066fcb3b810aa8826da0841385913646cfa29a4"} Oct 29 22:48:13.751571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:13.751515 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" podStartSLOduration=1.7514998149999998 podStartE2EDuration="1.751499815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:13.746256406 +0000 UTC m=+1960.640331330" watchObservedRunningTime="2026-10-29 22:48:13.751499815 +0000 UTC m=+1960.645574735" Oct 29 22:48:14.073723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:14.072096 5367 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 29 22:48:14.073723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:14.072155 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 29 22:48:14.716024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:14.715991 5367 generic.go:356] "Generic (PLEG): container finished" podID="1786c820-4fda-4b71-9490-d207be700219" containerID="ead33f1907994949fcdef6c1b3e719824fd77f44171b1416208497fe5386ea8f" exitCode=0 Oct 29 22:48:14.716211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:14.716078 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" event={"ID":"1786c820-4fda-4b71-9490-d207be700219","Type":"ContainerDied","Data":"ead33f1907994949fcdef6c1b3e719824fd77f44171b1416208497fe5386ea8f"} Oct 29 22:48:15.071736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:15.068104 5367 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 29 22:48:15.071736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:15.068165 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 29 22:48:15.970624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:15.970580 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:16.079725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.068294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca\") pod \"1786c820-4fda-4b71-9490-d207be700219\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " Oct 29 22:48:16.079725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.068596 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca\") pod \"1786c820-4fda-4b71-9490-d207be700219\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " Oct 29 22:48:16.079725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.068733 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm\" (UniqueName: \"kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm\") pod \"1786c820-4fda-4b71-9490-d207be700219\" (UID: \"1786c820-4fda-4b71-9490-d207be700219\") " Oct 29 22:48:16.081823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.081686 5367 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 29 22:48:16.082963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.081913 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca" pod "1786c820-4fda-4b71-9490-d207be700219" (UID: "1786c820-4fda-4b71-9490-d207be700219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:16.083087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.081856 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 29 22:48:16.083738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.083663 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca" pod "1786c820-4fda-4b71-9490-d207be700219" (UID: "1786c820-4fda-4b71-9490-d207be700219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:16.084305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.084279 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm" pod "1786c820-4fda-4b71-9490-d207be700219" (UID: "1786c820-4fda-4b71-9490-d207be700219"). InnerVolumeSpecName "kube-api-access-tvrqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:16.178767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.172171 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1786c820-4fda-4b71-9490-d207be700219-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:16.178767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.172214 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrqm\" (UniqueName: \"kubernetes.io/projected/1786c820-4fda-4b71-9490-d207be700219-kube-api-access-tvrqm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:16.178767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.172232 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1786c820-4fda-4b71-9490-d207be700219-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:16.735587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.735557 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" Oct 29 22:48:16.735821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.735586 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm" event={"ID":"1786c820-4fda-4b71-9490-d207be700219","Type":"ContainerDied","Data":"a0059c9b88308f1d312b5a856066fcb3b810aa8826da0841385913646cfa29a4"} Oct 29 22:48:16.735821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:16.735656 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0059c9b88308f1d312b5a856066fcb3b810aa8826da0841385913646cfa29a4" Oct 29 22:48:17.071662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:17.071634 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:48:17.071996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:17.071975 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:48:17.072660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:17.072643 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:48:22.333910 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:48:22.333867 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:48:25.762310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.762275 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6"] Oct 29 22:48:25.763690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.763663 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1786c820-4fda-4b71-9490-d207be700219" containerName="revoke-keys" Oct 29 22:48:25.763690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.763689 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1786c820-4fda-4b71-9490-d207be700219" containerName="revoke-keys" Oct 29 22:48:25.763954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.763940 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1786c820-4fda-4b71-9490-d207be700219" containerName="revoke-keys" Oct 29 22:48:25.767906 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.767887 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.787503 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.787477 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6"] Oct 29 22:48:25.794283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.794255 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:25.794283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.794260 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:48:25.895586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.895552 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.895827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.895806 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.895877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.895841 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798l6\" (UniqueName: \"kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.997435 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.997403 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.997617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.997443 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798l6\" (UniqueName: \"kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.997669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.997624 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.998305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.998273 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:25.999970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:25.999946 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:26.029340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:26.029273 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-798l6\" (UniqueName: \"kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6\") pod \"maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:26.084645 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:26.084612 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:26.397026 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:48:26.396992 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585589e4_16ac_48b7_9665_5c795d4b04b3.slice/crio-f645f1a7db235e7e49dcc5bb5bf84277a70271a6184eae3aec2b303b124244ac WatchSource:0}: Error finding container f645f1a7db235e7e49dcc5bb5bf84277a70271a6184eae3aec2b303b124244ac: Status 404 returned error can't find the container with id f645f1a7db235e7e49dcc5bb5bf84277a70271a6184eae3aec2b303b124244ac Oct 29 22:48:26.397395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:26.397375 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6"] Oct 29 22:48:26.867086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:26.867044 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" event={"ID":"585589e4-16ac-48b7-9665-5c795d4b04b3","Type":"ContainerStarted","Data":"3ea7cbf8c230fc183a6a1c0e84729f1f3f954125e52161a77113f49004142f1f"} Oct 29 22:48:26.867573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:26.867111 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" event={"ID":"585589e4-16ac-48b7-9665-5c795d4b04b3","Type":"ContainerStarted","Data":"f645f1a7db235e7e49dcc5bb5bf84277a70271a6184eae3aec2b303b124244ac"} Oct 29 22:48:26.928722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:26.919625 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" podStartSLOduration=1.919605631 podStartE2EDuration="1.919605631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:26.917459618 +0000 UTC m=+1973.811534541" watchObservedRunningTime="2026-10-29 22:48:26.919605631 +0000 UTC m=+1973.813680553" Oct 29 22:48:27.878020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:27.877988 5367 generic.go:356] "Generic (PLEG): container finished" podID="585589e4-16ac-48b7-9665-5c795d4b04b3" containerID="3ea7cbf8c230fc183a6a1c0e84729f1f3f954125e52161a77113f49004142f1f" exitCode=0 Oct 29 22:48:27.878422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:27.878054 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" event={"ID":"585589e4-16ac-48b7-9665-5c795d4b04b3","Type":"ContainerDied","Data":"3ea7cbf8c230fc183a6a1c0e84729f1f3f954125e52161a77113f49004142f1f"} Oct 29 22:48:29.060980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.060951 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:29.159869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.159768 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca\") pod \"585589e4-16ac-48b7-9665-5c795d4b04b3\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " Oct 29 22:48:29.160063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.159905 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6\" (UniqueName: \"kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6\") pod \"585589e4-16ac-48b7-9665-5c795d4b04b3\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " Oct 29 22:48:29.160200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.160179 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca\") pod \"585589e4-16ac-48b7-9665-5c795d4b04b3\" (UID: \"585589e4-16ac-48b7-9665-5c795d4b04b3\") " Oct 29 22:48:29.160510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.160485 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca" pod "585589e4-16ac-48b7-9665-5c795d4b04b3" (UID: "585589e4-16ac-48b7-9665-5c795d4b04b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:29.160983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.160959 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:29.162492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.162464 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca" pod "585589e4-16ac-48b7-9665-5c795d4b04b3" (UID: "585589e4-16ac-48b7-9665-5c795d4b04b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:29.162605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.162485 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6" pod "585589e4-16ac-48b7-9665-5c795d4b04b3" (UID: "585589e4-16ac-48b7-9665-5c795d4b04b3"). InnerVolumeSpecName "kube-api-access-798l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:29.262665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.262630 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585589e4-16ac-48b7-9665-5c795d4b04b3-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:29.262665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.262660 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-798l6\" (UniqueName: \"kubernetes.io/projected/585589e4-16ac-48b7-9665-5c795d4b04b3-kube-api-access-798l6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:29.896758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.896731 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" Oct 29 22:48:29.896758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.896752 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6" event={"ID":"585589e4-16ac-48b7-9665-5c795d4b04b3","Type":"ContainerDied","Data":"f645f1a7db235e7e49dcc5bb5bf84277a70271a6184eae3aec2b303b124244ac"} Oct 29 22:48:29.896984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:29.896800 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f645f1a7db235e7e49dcc5bb5bf84277a70271a6184eae3aec2b303b124244ac" Oct 29 22:48:43.651616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:43.651581 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6"] Oct 29 22:48:43.664296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:43.664270 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a558b3-pmwn6"] Oct 29 22:48:45.225602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:45.225528 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585589e4-16ac-48b7-9665-5c795d4b04b3" path="/var/lib/kubelet/pods/585589e4-16ac-48b7-9665-5c795d4b04b3/volumes" Oct 29 22:48:51.617826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:51.617791 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4"] Oct 29 22:48:51.618653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:51.618618 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" containerID="cri-o://cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873" gracePeriod=30 Oct 29 22:48:52.154807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.154771 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5"] Oct 29 22:48:52.155665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.155648 5367 cpu_manager.go:414] "containerMap: removing container" podUID="585589e4-16ac-48b7-9665-5c795d4b04b3" containerName="revoke-keys" Oct 29 22:48:52.155727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.155668 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585589e4-16ac-48b7-9665-5c795d4b04b3" containerName="revoke-keys" Oct 29 22:48:52.155828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.155818 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="585589e4-16ac-48b7-9665-5c795d4b04b3" containerName="revoke-keys" Oct 29 22:48:52.161223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.161200 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.188801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.188764 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5"] Oct 29 22:48:52.193182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.193154 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:52.193360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.193191 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:48:52.254402 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.254357 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.254668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.254644 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdwz\" (UniqueName: \"kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.254879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.254855 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.333825 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:48:52.333789 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:48:52.356968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.356938 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdwz\" (UniqueName: \"kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.357116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.357032 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.357169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.357135 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.357776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.357752 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.359355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.359329 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.377152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.377125 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdwz\" (UniqueName: \"kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz\") pod \"maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.476169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.476103 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:52.787244 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:52.787206 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5"] Oct 29 22:48:53.254711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:53.254653 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" event={"ID":"91c69f6f-fb2a-4eec-bc10-ec8f795dd260","Type":"ContainerStarted","Data":"12d20833c45e3e67b9d2d4ab83d3cfe1d744ef6ec6a145075df6a1208fb9ffe7"} Oct 29 22:48:53.254893 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:53.254749 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" event={"ID":"91c69f6f-fb2a-4eec-bc10-ec8f795dd260","Type":"ContainerStarted","Data":"5b5f74da556d4ed23490f66ebcb4c4f955b5427da4d26102f39715623bbcfa05"} Oct 29 22:48:53.298613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:53.298554 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" podStartSLOduration=1.298541128 podStartE2EDuration="1.298541128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:53.296369451 +0000 UTC m=+2000.190444374" watchObservedRunningTime="2026-10-29 22:48:53.298541128 +0000 UTC m=+2000.192616048" Oct 29 22:48:54.280072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:54.280024 5367 generic.go:356] "Generic (PLEG): container finished" podID="91c69f6f-fb2a-4eec-bc10-ec8f795dd260" containerID="12d20833c45e3e67b9d2d4ab83d3cfe1d744ef6ec6a145075df6a1208fb9ffe7" exitCode=0 Oct 29 22:48:54.280558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:54.280219 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" event={"ID":"91c69f6f-fb2a-4eec-bc10-ec8f795dd260","Type":"ContainerDied","Data":"12d20833c45e3e67b9d2d4ab83d3cfe1d744ef6ec6a145075df6a1208fb9ffe7"} Oct 29 22:48:55.458760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.458724 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:55.549901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.549812 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca\") pod \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " Oct 29 22:48:55.550080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.549961 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca\") pod \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " Oct 29 22:48:55.550080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.550005 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz\" (UniqueName: \"kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz\") pod \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\" (UID: \"91c69f6f-fb2a-4eec-bc10-ec8f795dd260\") " Oct 29 22:48:55.551003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.550976 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca" pod "91c69f6f-fb2a-4eec-bc10-ec8f795dd260" (UID: "91c69f6f-fb2a-4eec-bc10-ec8f795dd260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:55.551291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.551271 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:55.561156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.561133 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca" pod "91c69f6f-fb2a-4eec-bc10-ec8f795dd260" (UID: "91c69f6f-fb2a-4eec-bc10-ec8f795dd260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:55.561286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.561207 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz" pod "91c69f6f-fb2a-4eec-bc10-ec8f795dd260" (UID: "91c69f6f-fb2a-4eec-bc10-ec8f795dd260"). InnerVolumeSpecName "kube-api-access-nzdwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:55.653522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.653493 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:55.653522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:55.653518 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzdwz\" (UniqueName: \"kubernetes.io/projected/91c69f6f-fb2a-4eec-bc10-ec8f795dd260-kube-api-access-nzdwz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:56.299810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.299777 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" Oct 29 22:48:56.299810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.299791 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5" event={"ID":"91c69f6f-fb2a-4eec-bc10-ec8f795dd260","Type":"ContainerDied","Data":"5b5f74da556d4ed23490f66ebcb4c4f955b5427da4d26102f39715623bbcfa05"} Oct 29 22:48:56.300035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.299845 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5f74da556d4ed23490f66ebcb4c4f955b5427da4d26102f39715623bbcfa05" Oct 29 22:48:56.913763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.913733 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:48:56.976648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.976612 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.976880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.976809 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.976956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.976945 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977016 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977113 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977182 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977237 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977274 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977348 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo\") pod \"90549b69-0a71-4e73-9d2c-f9792b2aa082\" (UID: \"90549b69-0a71-4e73-9d2c-f9792b2aa082\") " Oct 29 22:48:56.977759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977381 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:56.977759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977398 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:56.977759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977583 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:56.977759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977639 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:56.978000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.977812 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:56.978292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.978264 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:56.978416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.978324 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:56.978416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.978341 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:56.978416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.978356 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:56.978416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.978370 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90549b69-0a71-4e73-9d2c-f9792b2aa082-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:56.980074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.980046 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "kube-api-access-2z44z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:56.980200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.980151 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:56.980950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.980922 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:56.981922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:56.981896 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy" pod "90549b69-0a71-4e73-9d2c-f9792b2aa082" (UID: "90549b69-0a71-4e73-9d2c-f9792b2aa082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:57.080902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.080867 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.080902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.080905 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.081137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.080919 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90549b69-0a71-4e73-9d2c-f9792b2aa082-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.081137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.080933 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z44z\" (UniqueName: \"kubernetes.io/projected/90549b69-0a71-4e73-9d2c-f9792b2aa082-kube-api-access-2z44z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.309601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.309568 5367 generic.go:356] "Generic (PLEG): container finished" podID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerID="cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873" exitCode=0 Oct 29 22:48:57.309816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.309616 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" Oct 29 22:48:57.309816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.309637 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" event={"ID":"90549b69-0a71-4e73-9d2c-f9792b2aa082","Type":"ContainerDied","Data":"cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873"} Oct 29 22:48:57.309816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.309739 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4" event={"ID":"90549b69-0a71-4e73-9d2c-f9792b2aa082","Type":"ContainerDied","Data":"f07fea3161fd58f2a153ff24fdab52484c0afa7be2b42eace4f759cb48ae8f00"} Oct 29 22:48:57.309816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.309742 5367 scope.go:122] "RemoveContainer" containerID="cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873" Oct 29 22:48:57.321806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.321785 5367 scope.go:122] "RemoveContainer" containerID="cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873" Oct 29 22:48:57.322128 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:48:57.322105 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873\": container with ID starting with cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873 not found: ID does not exist" containerID="cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873" Oct 29 22:48:57.322208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.322138 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873"} err="failed to get container status \"cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873\": rpc error: code = NotFound desc = could not find container \"cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873\": container with ID starting with cf0e4717fe7153b4fa4b4c61b201d5557f5844f54c21a47e4b01daef463be873 not found: ID does not exist" Oct 29 22:48:57.368745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.368714 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4"] Oct 29 22:48:57.381279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:57.381252 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-4vmt4"] Oct 29 22:48:59.226036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:48:59.226004 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" path="/var/lib/kubelet/pods/90549b69-0a71-4e73-9d2c-f9792b2aa082/volumes" Oct 29 22:49:03.441392 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.441342 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn"] Oct 29 22:49:03.460994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.460960 5367 cpu_manager.go:414] "containerMap: removing container" podUID="91c69f6f-fb2a-4eec-bc10-ec8f795dd260" containerName="revoke-keys" Oct 29 22:49:03.461235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.461221 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c69f6f-fb2a-4eec-bc10-ec8f795dd260" containerName="revoke-keys" Oct 29 22:49:03.461331 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.461322 5367 cpu_manager.go:414] "containerMap: removing container" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" Oct 29 22:49:03.461414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.461404 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" Oct 29 22:49:03.461855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.461838 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c69f6f-fb2a-4eec-bc10-ec8f795dd260" containerName="revoke-keys" Oct 29 22:49:03.461981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.461970 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="90549b69-0a71-4e73-9d2c-f9792b2aa082" containerName="istio-proxy" Oct 29 22:49:03.483049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.479990 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn"] Oct 29 22:49:03.483049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.480122 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.496538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.496510 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ad69b4-openshift-default-dockercfg-mltld\"" Oct 29 22:49:03.599792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.599758 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.599991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.599890 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.599991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.599927 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.599991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.599970 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8cn\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.600135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.600036 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.600135 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.600084 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.600211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.600185 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.600291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.600276 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.600384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.600371 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701260 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701307 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701391 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701494 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701551 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701694 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701783 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701850 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701854 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8cn\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701931 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.701984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.701965 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.702219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.702033 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.702219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.702108 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.703814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.703791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.704310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.704293 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.721864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.721836 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.722257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.722237 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8cn\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:03.809110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:03.809080 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:04.136613 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:49:04.136579 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275ee01a_c16a_42f8_b332_e2728b091f71.slice/crio-0b7dd74d7eca88f49f1f407a41f6abb528ba12fe48427169f0c63dfa50f8f20e WatchSource:0}: Error finding container 0b7dd74d7eca88f49f1f407a41f6abb528ba12fe48427169f0c63dfa50f8f20e: Status 404 returned error can't find the container with id 0b7dd74d7eca88f49f1f407a41f6abb528ba12fe48427169f0c63dfa50f8f20e Oct 29 22:49:04.136780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.136667 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn"] Oct 29 22:49:04.139522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.139490 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:04.139606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.139567 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:04.139643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.139608 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:04.545825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.544599 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" event={"ID":"275ee01a-c16a-42f8-b332-e2728b091f71","Type":"ContainerStarted","Data":"afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75"} Oct 29 22:49:04.545825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.544732 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" event={"ID":"275ee01a-c16a-42f8-b332-e2728b091f71","Type":"ContainerStarted","Data":"0b7dd74d7eca88f49f1f407a41f6abb528ba12fe48427169f0c63dfa50f8f20e"} Oct 29 22:49:04.647037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.646970 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podStartSLOduration=1.646924907 podStartE2EDuration="1.646924907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:04.59974224 +0000 UTC m=+2011.493817162" watchObservedRunningTime="2026-10-29 22:49:04.646924907 +0000 UTC m=+2011.540999831" Oct 29 22:49:04.809739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:04.809685 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:05.811001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:05.810965 5367 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 29 22:49:05.811462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:05.811021 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 29 22:49:06.810459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:06.810414 5367 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 29 22:49:06.810683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:06.810482 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 29 22:49:07.809865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:07.809828 5367 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 29 22:49:07.810260 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:07.809889 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 29 22:49:08.809553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:08.809517 5367 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 29 22:49:08.809792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:08.809576 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 29 22:49:09.817038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:09.817004 5367 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 29 22:49:09.817444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:09.817198 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 29 22:49:10.809468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:10.809424 5367 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 29 22:49:10.809687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:10.809488 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 29 22:49:11.814835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:11.814802 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:11.815314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:11.815209 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:11.815834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:11.815815 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:49:14.705500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:14.705467 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn"] Oct 29 22:49:14.706007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:14.705744 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" podUID="873f0eaf-50ea-4a57-8f20-cb6034928af5" containerName="maas-api" containerID="cri-o://3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b" gracePeriod=30 Oct 29 22:49:15.016638 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.016612 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:49:15.124780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.119475 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls\") pod \"873f0eaf-50ea-4a57-8f20-cb6034928af5\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " Oct 29 22:49:15.124780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.119557 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs\") pod \"873f0eaf-50ea-4a57-8f20-cb6034928af5\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " Oct 29 22:49:15.124780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.119799 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r\" (UniqueName: \"kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r\") pod \"873f0eaf-50ea-4a57-8f20-cb6034928af5\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " Oct 29 22:49:15.124780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.119855 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca\") pod \"873f0eaf-50ea-4a57-8f20-cb6034928af5\" (UID: \"873f0eaf-50ea-4a57-8f20-cb6034928af5\") " Oct 29 22:49:15.167725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.156937 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca" pod "873f0eaf-50ea-4a57-8f20-cb6034928af5" (UID: "873f0eaf-50ea-4a57-8f20-cb6034928af5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:15.175673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.175305 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs" pod "873f0eaf-50ea-4a57-8f20-cb6034928af5" (UID: "873f0eaf-50ea-4a57-8f20-cb6034928af5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:49:15.187754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.183861 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls" pod "873f0eaf-50ea-4a57-8f20-cb6034928af5" (UID: "873f0eaf-50ea-4a57-8f20-cb6034928af5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:49:15.234197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.234031 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r" pod "873f0eaf-50ea-4a57-8f20-cb6034928af5" (UID: "873f0eaf-50ea-4a57-8f20-cb6034928af5"). InnerVolumeSpecName "kube-api-access-w5m4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:15.265994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.256977 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:15.270747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.270650 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873f0eaf-50ea-4a57-8f20-cb6034928af5-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:15.271000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.270986 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5m4r\" (UniqueName: \"kubernetes.io/projected/873f0eaf-50ea-4a57-8f20-cb6034928af5-kube-api-access-w5m4r\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:15.271122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.271096 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873f0eaf-50ea-4a57-8f20-cb6034928af5-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:15.591286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.591252 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5"] Oct 29 22:49:15.603578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.603550 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-139eb7-s4cc5"] Oct 29 22:49:15.697011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.696979 5367 generic.go:356] "Generic (PLEG): container finished" podID="873f0eaf-50ea-4a57-8f20-cb6034928af5" containerID="3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b" exitCode=0 Oct 29 22:49:15.697178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.697020 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" Oct 29 22:49:15.697178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.697060 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" event={"ID":"873f0eaf-50ea-4a57-8f20-cb6034928af5","Type":"ContainerDied","Data":"3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b"} Oct 29 22:49:15.697178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.697112 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn" event={"ID":"873f0eaf-50ea-4a57-8f20-cb6034928af5","Type":"ContainerDied","Data":"564ee0e93ae8623fe94927636c129cb5046fd491d50ae7b86b3736f9b37f2e07"} Oct 29 22:49:15.697178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.697136 5367 scope.go:122] "RemoveContainer" containerID="3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b" Oct 29 22:49:15.709218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.709041 5367 scope.go:122] "RemoveContainer" containerID="3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b" Oct 29 22:49:15.709471 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:49:15.709309 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b\": container with ID starting with 3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b not found: ID does not exist" containerID="3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b" Oct 29 22:49:15.709471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.709336 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b"} err="failed to get container status \"3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b\": rpc error: code = NotFound desc = could not find container \"3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b\": container with ID starting with 3cf466d791979e840b21a11076074e4dd9570cbf262aae36434338c53a56953b not found: ID does not exist" Oct 29 22:49:15.787455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.787415 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn"] Oct 29 22:49:15.792867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:15.792841 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-139eb7-c46566579-j2gfn"] Oct 29 22:49:17.224464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:17.224429 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873f0eaf-50ea-4a57-8f20-cb6034928af5" path="/var/lib/kubelet/pods/873f0eaf-50ea-4a57-8f20-cb6034928af5/volumes" Oct 29 22:49:17.224938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:17.224922 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c69f6f-fb2a-4eec-bc10-ec8f795dd260" path="/var/lib/kubelet/pods/91c69f6f-fb2a-4eec-bc10-ec8f795dd260/volumes" Oct 29 22:49:22.333045 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:49:22.333002 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:49:24.320841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:24.320802 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd"] Oct 29 22:49:24.321344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:24.321050 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" containerID="cri-o://a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7" gracePeriod=30 Oct 29 22:49:29.629106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.629077 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:49:29.724470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724436 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.724667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724489 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.724667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724521 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.724667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724625 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.724877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724807 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.724877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724870 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.724991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724931 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.724991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.724971 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.725107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.725093 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.725162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.725143 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.725221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.725181 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo\") pod \"3624b108-b174-4506-9d31-76991ddb24d8\" (UID: \"3624b108-b174-4506-9d31-76991ddb24d8\") " Oct 29 22:49:29.725479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.725413 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.725479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.725452 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.725877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.725851 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:29.726119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.726096 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.726119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.726123 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.726286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.726139 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.726286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.726153 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.727739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.727639 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:49:29.727831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.727742 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "kube-api-access-p86xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:29.727831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.727791 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:29.729404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.729383 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy" pod "3624b108-b174-4506-9d31-76991ddb24d8" (UID: "3624b108-b174-4506-9d31-76991ddb24d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.824811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.824775 5367 generic.go:356] "Generic (PLEG): container finished" podID="3624b108-b174-4506-9d31-76991ddb24d8" containerID="a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7" exitCode=0 Oct 29 22:49:29.825000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.824818 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" event={"ID":"3624b108-b174-4506-9d31-76991ddb24d8","Type":"ContainerDied","Data":"a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7"} Oct 29 22:49:29.825000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.824875 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" event={"ID":"3624b108-b174-4506-9d31-76991ddb24d8","Type":"ContainerDied","Data":"3369846aae8260a72ad18e14f12a822b2e641c7d636202a048959f5472bc7e0e"} Oct 29 22:49:29.825000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.824900 5367 scope.go:122] "RemoveContainer" containerID="a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7" Oct 29 22:49:29.825000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.824831 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd" Oct 29 22:49:29.827955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.827928 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86xq\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-kube-api-access-p86xq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.828089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.827959 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3624b108-b174-4506-9d31-76991ddb24d8-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.828089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.827974 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3624b108-b174-4506-9d31-76991ddb24d8-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.828089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.827987 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3624b108-b174-4506-9d31-76991ddb24d8-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.828089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.828000 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3624b108-b174-4506-9d31-76991ddb24d8-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.840831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.840812 5367 scope.go:122] "RemoveContainer" containerID="a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7" Oct 29 22:49:29.841102 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:49:29.841082 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7\": container with ID starting with a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7 not found: ID does not exist" containerID="a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7" Oct 29 22:49:29.841162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.841110 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7"} err="failed to get container status \"a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7\": rpc error: code = NotFound desc = could not find container \"a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7\": container with ID starting with a070d8b5032c390b505b4de093cb8dace36695cc4d64b3963cac605a1ba876c7 not found: ID does not exist" Oct 29 22:49:29.881992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.881955 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd"] Oct 29 22:49:29.886770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:29.886743 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-k97cd"] Oct 29 22:49:31.225997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:31.225964 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3624b108-b174-4506-9d31-76991ddb24d8" path="/var/lib/kubelet/pods/3624b108-b174-4506-9d31-76991ddb24d8/volumes" Oct 29 22:49:37.059897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:37.059863 5367 scope.go:122] "RemoveContainer" containerID="4755850e2c465a597648af39d6dcca0e179d3ec934b0f39631756da6fc30c486" Oct 29 22:49:38.030712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:38.030676 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4"] Oct 29 22:49:38.043554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:38.043526 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc47886c-vdwp4"] Oct 29 22:49:39.240521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:39.240485 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3305448f-5002-4224-afb8-8b1c50294f21" path="/var/lib/kubelet/pods/3305448f-5002-4224-afb8-8b1c50294f21/volumes" Oct 29 22:49:45.396725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:45.395827 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" podUID="3b64379c-50a1-4938-ac56-5a49ab326919" containerName="main" probeResult="failure" output="Get \"https://10.133.0.246:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:49:45.637723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:45.636832 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 29 22:49:52.336283 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:49:52.336244 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:49:53.659861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:53.659820 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:49:53.660264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:53.659892 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:49:55.177477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:55.177443 5367 generic.go:356] "Generic (PLEG): container finished" podID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerID="fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d" exitCode=0 Oct 29 22:49:55.177928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:55.177508 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" event={"ID":"0e7d7e47-2202-4ea4-adc7-2804d703c55e","Type":"ContainerDied","Data":"fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d"} Oct 29 22:49:55.178003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:55.177988 5367 scope.go:122] "RemoveContainer" containerID="fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d" Oct 29 22:49:55.180318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:55.180284 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:55.180386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:55.180372 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:55.180430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:55.180415 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:56.191600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:56.191566 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" event={"ID":"0e7d7e47-2202-4ea4-adc7-2804d703c55e","Type":"ContainerStarted","Data":"63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1"} Oct 29 22:49:56.711069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:56.711033 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:49:56.711361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:56.711343 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:49:56.711410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:56.711394 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 29 22:49:57.711032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:57.710991 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:49:57.711465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:57.711054 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 29 22:49:58.711384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:58.711347 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:49:58.711833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:58.711403 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 29 22:49:59.712719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:59.711106 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:49:59.712719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:49:59.711170 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 29 22:50:00.711111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:00.711077 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:50:00.711439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:00.711144 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 29 22:50:01.753046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:01.752892 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:50:01.753762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:01.753735 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:50:01.754587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:01.754563 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:50:09.035240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.035206 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv"] Oct 29 22:50:09.036120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.036102 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" Oct 29 22:50:09.036185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.036121 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" Oct 29 22:50:09.036185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.036138 5367 cpu_manager.go:414] "containerMap: removing container" podUID="873f0eaf-50ea-4a57-8f20-cb6034928af5" containerName="maas-api" Oct 29 22:50:09.036185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.036145 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873f0eaf-50ea-4a57-8f20-cb6034928af5" containerName="maas-api" Oct 29 22:50:09.036288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.036281 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3624b108-b174-4506-9d31-76991ddb24d8" containerName="istio-proxy" Oct 29 22:50:09.036323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.036292 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="873f0eaf-50ea-4a57-8f20-cb6034928af5" containerName="maas-api" Oct 29 22:50:09.040067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.040046 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.056751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.056725 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-437981-openshift-default-dockercfg-d74jz\"" Oct 29 22:50:09.077815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.077782 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv"] Oct 29 22:50:09.144708 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.144670 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.144896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.144813 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.144896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.144853 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.144896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.144880 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.145086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.144951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.145146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.145078 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.145219 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.145200 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrgw\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.145282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.145271 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.145343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.145319 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.246615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.246578 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrgw\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.246933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.246625 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.246933 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.246656 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.246953 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247118 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247225 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247291 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247347 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247402 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247458 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247746 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247824 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.247988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.247948 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.248176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.248150 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.249929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.249908 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.250013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.249947 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.267250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.267219 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrgw\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.267340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.267261 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.354810 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.354758 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:09.692851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.692818 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv"] Oct 29 22:50:09.693098 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:50:09.693077 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251fc4be_6212_4e18_9759_8802ee120fd0.slice/crio-a170ba110c1a66d6f50e45e4f001608d2311f36a64b94bf9589ec5539d502f78 WatchSource:0}: Error finding container a170ba110c1a66d6f50e45e4f001608d2311f36a64b94bf9589ec5539d502f78: Status 404 returned error can't find the container with id a170ba110c1a66d6f50e45e4f001608d2311f36a64b94bf9589ec5539d502f78 Oct 29 22:50:09.696094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.696060 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:50:09.696174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.696129 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:50:09.696174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:09.696160 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:50:10.387874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:10.387833 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" event={"ID":"251fc4be-6212-4e18-9759-8802ee120fd0","Type":"ContainerStarted","Data":"4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa"} Oct 29 22:50:10.388313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:10.387899 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" event={"ID":"251fc4be-6212-4e18-9759-8802ee120fd0","Type":"ContainerStarted","Data":"a170ba110c1a66d6f50e45e4f001608d2311f36a64b94bf9589ec5539d502f78"} Oct 29 22:50:10.478727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:10.477691 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podStartSLOduration=1.477670732 podStartE2EDuration="1.477670732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:10.445122758 +0000 UTC m=+2077.339197679" watchObservedRunningTime="2026-10-29 22:50:10.477670732 +0000 UTC m=+2077.371745656" Oct 29 22:50:11.378766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:11.377836 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:11.380762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:11.380728 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 29 22:50:11.380903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:11.380799 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 29 22:50:12.378721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:12.376812 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 29 22:50:12.378721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:12.376887 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 29 22:50:13.361725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:13.359074 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 29 22:50:13.361725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:13.359138 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 29 22:50:14.379296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:14.379204 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 29 22:50:14.379296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:14.379280 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 29 22:50:15.034205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:15.034168 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh"] Oct 29 22:50:15.047222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:15.047193 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a650f20-cqtvh"] Oct 29 22:50:15.226583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:15.226546 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253608fc-b2c2-4258-ad1f-3eb17f558fd8" path="/var/lib/kubelet/pods/253608fc-b2c2-4258-ad1f-3eb17f558fd8/volumes" Oct 29 22:50:15.371722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:15.366097 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 29 22:50:15.371722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:15.366159 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 29 22:50:16.360862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:16.360830 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:16.361341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:16.361328 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:16.362021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:16.361999 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 22:50:22.333126 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:50:22.333088 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:50:23.530924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.530890 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4"] Oct 29 22:50:23.536825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.536793 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.564631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.564600 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"auth-test-model-dbb507-kserve-self-signed-certs\"" Oct 29 22:50:23.564848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.564798 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"default-dockercfg-nn5ll\"" Oct 29 22:50:23.567953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.567927 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"openshift-service-ca.crt\"" Oct 29 22:50:23.568103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.568023 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"kube-root-ca.crt\"" Oct 29 22:50:23.571951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.571798 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4"] Oct 29 22:50:23.642520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.642475 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.642839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.642817 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.642945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.642872 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.642992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.642938 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcht\" (UniqueName: \"kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.643041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.643017 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.643095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.643081 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.744792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.744743 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.744888 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745122 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745213 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745169 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745241 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcht\" (UniqueName: \"kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745264 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745293 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745491 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.745675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.745659 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.747330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.747305 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.747457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.747389 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.767044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.767017 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcht\" (UniqueName: \"kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht\") pod \"auth-test-model-dbb507-kserve-9598d99cc-ddwv4\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:23.860060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:23.860018 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:24.225920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:24.225889 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4"] Oct 29 22:50:24.226043 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:50:24.225946 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0b9ea9c_b8b7_4c2b_aec2_a14e941a28fd.slice/crio-bd4fbc5ef6d5aea208bdc97f78f6444663ca3bd92db30bee6b4ffe883f7c8126 WatchSource:0}: Error finding container bd4fbc5ef6d5aea208bdc97f78f6444663ca3bd92db30bee6b4ffe883f7c8126: Status 404 returned error can't find the container with id bd4fbc5ef6d5aea208bdc97f78f6444663ca3bd92db30bee6b4ffe883f7c8126 Oct 29 22:50:24.530406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:24.530310 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" event={"ID":"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd","Type":"ContainerStarted","Data":"455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5"} Oct 29 22:50:24.530406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:24.530355 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" event={"ID":"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd","Type":"ContainerStarted","Data":"bd4fbc5ef6d5aea208bdc97f78f6444663ca3bd92db30bee6b4ffe883f7c8126"} Oct 29 22:50:24.575966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:24.575904 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" podStartSLOduration=1.5758905749999998 podStartE2EDuration="1.575890575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:24.57383382 +0000 UTC m=+2091.467908755" watchObservedRunningTime="2026-10-29 22:50:24.575890575 +0000 UTC m=+2091.469965487" Oct 29 22:50:24.861002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:24.860972 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:29.087437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.087401 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt"] Oct 29 22:50:29.092388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.092363 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.108204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.108008 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-6bb06a\"/\"openshift-service-ca.crt\"" Oct 29 22:50:29.116644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.116614 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt"] Oct 29 22:50:29.118182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.118157 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-6bb06a\"/\"kube-root-ca.crt\"" Oct 29 22:50:29.118364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.118286 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-6bb06a\"/\"default-dockercfg-xjbzs\"" Oct 29 22:50:29.121131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.121102 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-6bb06a\"/\"sub-test-model-6bb06a-kserve-self-signed-certs\"" Oct 29 22:50:29.133072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.133045 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.133194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.133100 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.133309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.133282 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrdp\" (UniqueName: \"kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.133365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.133336 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.133433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.133415 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.133482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.133452 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235260 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235312 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235388 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrdp\" (UniqueName: \"kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235415 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235438 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235458 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235821 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235839 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.235948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.235932 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.237687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.237666 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.238029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.238010 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.255512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.255486 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrdp\" (UniqueName: \"kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp\") pod \"sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.416342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.416004 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:29.752619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:29.752594 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt"] Oct 29 22:50:29.753046 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:50:29.753024 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5649e738_d647_4ac7_a618_375f8aa91fc2.slice/crio-dfa63fa961d54ed7a0e6464625ff0284c1d9233c20fdea1d2a663c93f654e7f8 WatchSource:0}: Error finding container dfa63fa961d54ed7a0e6464625ff0284c1d9233c20fdea1d2a663c93f654e7f8: Status 404 returned error can't find the container with id dfa63fa961d54ed7a0e6464625ff0284c1d9233c20fdea1d2a663c93f654e7f8 Oct 29 22:50:30.596685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:30.596649 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" event={"ID":"5649e738-d647-4ac7-a618-375f8aa91fc2","Type":"ContainerStarted","Data":"c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb"} Oct 29 22:50:30.596685 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:30.596735 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" event={"ID":"5649e738-d647-4ac7-a618-375f8aa91fc2","Type":"ContainerStarted","Data":"dfa63fa961d54ed7a0e6464625ff0284c1d9233c20fdea1d2a663c93f654e7f8"} Oct 29 22:50:31.044853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:31.044746 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" podStartSLOduration=2.044731467 podStartE2EDuration="2.044731467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:30.639973882 +0000 UTC m=+2097.534048803" watchObservedRunningTime="2026-10-29 22:50:31.044731467 +0000 UTC m=+2097.938806388" Oct 29 22:50:31.045155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:31.045006 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt"] Oct 29 22:50:31.050693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:31.050666 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f870ae3-ppmwt"] Oct 29 22:50:31.227765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:31.227732 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23650b7-03fa-49f7-b2f9-2e75757c7a63" path="/var/lib/kubelet/pods/e23650b7-03fa-49f7-b2f9-2e75757c7a63/volumes" Oct 29 22:50:31.417184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:31.417153 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:33.922788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:33.922755 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:34.027040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:34.025268 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:34.872167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:34.872139 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:50:37.115579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:37.115546 5367 scope.go:122] "RemoveContainer" containerID="440c6e86f74f73cd42cd68f5d68511414d4897dc1b0e7f155a1feb28171b8f3c" Oct 29 22:50:37.127117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:37.127094 5367 scope.go:122] "RemoveContainer" containerID="dd7354b141fdd5f3434c876147b4394dfdc534fba6326293e4171e6e1ce0dcd1" Oct 29 22:50:39.349782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.349673 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n"] Oct 29 22:50:39.356367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.356348 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.374776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.374750 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n"] Oct 29 22:50:39.383091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.383064 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:50:39.383237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.383106 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:50:39.417161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.416924 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:39.434010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.433983 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:39.511486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.511453 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.511656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.511495 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.511656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.511639 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzbz\" (UniqueName: \"kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.612838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.612766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.612838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.612813 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.613031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.612874 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzbz\" (UniqueName: \"kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.613544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.613517 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.615088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.615065 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.633240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.633219 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzbz\" (UniqueName: \"kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz\") pod \"maas-api-revoke-sub-26f04bf9-njn7n\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.671498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.671472 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:39.932172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.932099 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:50:39.983985 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:50:39.983952 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb523e4d4_01c2_43c1_a454_31bd8734c455.slice/crio-a5a5ddca8a97cc05326daf8d3e807f333d00939c94c5b6207395937cef0fbf84 WatchSource:0}: Error finding container a5a5ddca8a97cc05326daf8d3e807f333d00939c94c5b6207395937cef0fbf84: Status 404 returned error can't find the container with id a5a5ddca8a97cc05326daf8d3e807f333d00939c94c5b6207395937cef0fbf84 Oct 29 22:50:39.985340 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:39.985314 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n"] Oct 29 22:50:40.931874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:40.931834 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" event={"ID":"b523e4d4-01c2-43c1-a454-31bd8734c455","Type":"ContainerStarted","Data":"18b3a278026f232fe85f63987771ea1e3da078018e0f18df7a559a70b552198c"} Oct 29 22:50:40.932375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:40.931898 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" event={"ID":"b523e4d4-01c2-43c1-a454-31bd8734c455","Type":"ContainerStarted","Data":"a5a5ddca8a97cc05326daf8d3e807f333d00939c94c5b6207395937cef0fbf84"} Oct 29 22:50:40.980819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:40.980764 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" podStartSLOduration=1.980746392 podStartE2EDuration="1.980746392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:40.972790714 +0000 UTC m=+2107.866865636" watchObservedRunningTime="2026-10-29 22:50:40.980746392 +0000 UTC m=+2107.874821314" Oct 29 22:50:41.035405 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:50:41.032058 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb523e4d4_01c2_43c1_a454_31bd8734c455.slice/crio-conmon-18b3a278026f232fe85f63987771ea1e3da078018e0f18df7a559a70b552198c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb523e4d4_01c2_43c1_a454_31bd8734c455.slice/crio-18b3a278026f232fe85f63987771ea1e3da078018e0f18df7a559a70b552198c.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:50:41.943672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:41.943639 5367 generic.go:356] "Generic (PLEG): container finished" podID="b523e4d4-01c2-43c1-a454-31bd8734c455" containerID="18b3a278026f232fe85f63987771ea1e3da078018e0f18df7a559a70b552198c" exitCode=0 Oct 29 22:50:41.944092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:41.943688 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" event={"ID":"b523e4d4-01c2-43c1-a454-31bd8734c455","Type":"ContainerDied","Data":"18b3a278026f232fe85f63987771ea1e3da078018e0f18df7a559a70b552198c"} Oct 29 22:50:43.129308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.129281 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:43.200242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.200205 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca\") pod \"b523e4d4-01c2-43c1-a454-31bd8734c455\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " Oct 29 22:50:43.200469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.200454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz\" (UniqueName: \"kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz\") pod \"b523e4d4-01c2-43c1-a454-31bd8734c455\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " Oct 29 22:50:43.200545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.200498 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca\") pod \"b523e4d4-01c2-43c1-a454-31bd8734c455\" (UID: \"b523e4d4-01c2-43c1-a454-31bd8734c455\") " Oct 29 22:50:43.200617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.200544 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca" pod "b523e4d4-01c2-43c1-a454-31bd8734c455" (UID: "b523e4d4-01c2-43c1-a454-31bd8734c455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:50:43.201174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.201147 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:50:43.202669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.202644 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca" pod "b523e4d4-01c2-43c1-a454-31bd8734c455" (UID: "b523e4d4-01c2-43c1-a454-31bd8734c455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:50:43.202786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.202751 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz" pod "b523e4d4-01c2-43c1-a454-31bd8734c455" (UID: "b523e4d4-01c2-43c1-a454-31bd8734c455"). InnerVolumeSpecName "kube-api-access-glzbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:50:43.302672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.302639 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzbz\" (UniqueName: \"kubernetes.io/projected/b523e4d4-01c2-43c1-a454-31bd8734c455-kube-api-access-glzbz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:50:43.302672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.302670 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b523e4d4-01c2-43c1-a454-31bd8734c455-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:50:43.966377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.966349 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" Oct 29 22:50:43.966377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.966369 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n" event={"ID":"b523e4d4-01c2-43c1-a454-31bd8734c455","Type":"ContainerDied","Data":"a5a5ddca8a97cc05326daf8d3e807f333d00939c94c5b6207395937cef0fbf84"} Oct 29 22:50:43.966622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:43.966429 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5a5ddca8a97cc05326daf8d3e807f333d00939c94c5b6207395937cef0fbf84" Oct 29 22:50:49.718999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.718961 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt"] Oct 29 22:50:49.720060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.720035 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b523e4d4-01c2-43c1-a454-31bd8734c455" containerName="revoke-keys" Oct 29 22:50:49.720060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.720059 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b523e4d4-01c2-43c1-a454-31bd8734c455" containerName="revoke-keys" Oct 29 22:50:49.720270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.720255 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b523e4d4-01c2-43c1-a454-31bd8734c455" containerName="revoke-keys" Oct 29 22:50:49.724512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.724492 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.744874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.744850 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt"] Oct 29 22:50:49.752732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.750743 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:50:49.752732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.750773 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:50:49.785281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.785251 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.785484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.785464 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.785655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.785632 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4m8\" (UniqueName: \"kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.887157 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.887131 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.887318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.887237 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.887318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.887306 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4m8\" (UniqueName: \"kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.887799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.887772 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.889552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.889530 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:49.908091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:49.908064 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4m8\" (UniqueName: \"kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8\") pod \"maas-api-revoke-sub-9220efd6-kxkxt\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:50.049780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:50.041253 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:50.371686 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:50:50.371654 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b02be5_8992_4244_803a_3a408aae05ae.slice/crio-f1d7b9cf18d5f6057d2ef62fd2a4e76e155cacc6430cf29917127c834952ac59 WatchSource:0}: Error finding container f1d7b9cf18d5f6057d2ef62fd2a4e76e155cacc6430cf29917127c834952ac59: Status 404 returned error can't find the container with id f1d7b9cf18d5f6057d2ef62fd2a4e76e155cacc6430cf29917127c834952ac59 Oct 29 22:50:50.374188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:50.374161 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt"] Oct 29 22:50:51.048166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:51.048074 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" event={"ID":"c1b02be5-8992-4244-803a-3a408aae05ae","Type":"ContainerStarted","Data":"b9d31bf5c9dc718095c046cdba639afdfdc26c96bea50184320838b89ba51a40"} Oct 29 22:50:51.048166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:51.048129 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" event={"ID":"c1b02be5-8992-4244-803a-3a408aae05ae","Type":"ContainerStarted","Data":"f1d7b9cf18d5f6057d2ef62fd2a4e76e155cacc6430cf29917127c834952ac59"} Oct 29 22:50:51.093352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:51.093302 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" podStartSLOduration=2.093288075 podStartE2EDuration="2.093288075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:51.089748722 +0000 UTC m=+2117.983823645" watchObservedRunningTime="2026-10-29 22:50:51.093288075 +0000 UTC m=+2117.987362995" Oct 29 22:50:52.058092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:52.058060 5367 generic.go:356] "Generic (PLEG): container finished" podID="c1b02be5-8992-4244-803a-3a408aae05ae" containerID="b9d31bf5c9dc718095c046cdba639afdfdc26c96bea50184320838b89ba51a40" exitCode=0 Oct 29 22:50:52.058517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:52.058174 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" event={"ID":"c1b02be5-8992-4244-803a-3a408aae05ae","Type":"ContainerDied","Data":"b9d31bf5c9dc718095c046cdba639afdfdc26c96bea50184320838b89ba51a40"} Oct 29 22:50:52.333365 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:50:52.333284 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:50:53.255582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.254969 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:53.353554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.353466 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca\") pod \"c1b02be5-8992-4244-803a-3a408aae05ae\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " Oct 29 22:50:53.353774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.353668 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8\" (UniqueName: \"kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8\") pod \"c1b02be5-8992-4244-803a-3a408aae05ae\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " Oct 29 22:50:53.353904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.353889 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca\") pod \"c1b02be5-8992-4244-803a-3a408aae05ae\" (UID: \"c1b02be5-8992-4244-803a-3a408aae05ae\") " Oct 29 22:50:53.354299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.354270 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca" pod "c1b02be5-8992-4244-803a-3a408aae05ae" (UID: "c1b02be5-8992-4244-803a-3a408aae05ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:50:53.354669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.354649 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:50:53.356009 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.355983 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8" pod "c1b02be5-8992-4244-803a-3a408aae05ae" (UID: "c1b02be5-8992-4244-803a-3a408aae05ae"). InnerVolumeSpecName "kube-api-access-lm4m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:50:53.356237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.356218 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca" pod "c1b02be5-8992-4244-803a-3a408aae05ae" (UID: "c1b02be5-8992-4244-803a-3a408aae05ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:50:53.456548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.456514 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4m8\" (UniqueName: \"kubernetes.io/projected/c1b02be5-8992-4244-803a-3a408aae05ae-kube-api-access-lm4m8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:50:53.456548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:53.456548 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b02be5-8992-4244-803a-3a408aae05ae-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:50:54.079622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:54.079589 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" Oct 29 22:50:54.079822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:54.079644 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt" event={"ID":"c1b02be5-8992-4244-803a-3a408aae05ae","Type":"ContainerDied","Data":"f1d7b9cf18d5f6057d2ef62fd2a4e76e155cacc6430cf29917127c834952ac59"} Oct 29 22:50:54.079822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:54.079690 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d7b9cf18d5f6057d2ef62fd2a4e76e155cacc6430cf29917127c834952ac59" Oct 29 22:50:58.525164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.525131 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp"] Oct 29 22:50:58.525968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.525953 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c1b02be5-8992-4244-803a-3a408aae05ae" containerName="revoke-keys" Oct 29 22:50:58.526022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.525969 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b02be5-8992-4244-803a-3a408aae05ae" containerName="revoke-keys" Oct 29 22:50:58.526119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.526109 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b02be5-8992-4244-803a-3a408aae05ae" containerName="revoke-keys" Oct 29 22:50:58.529970 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.529955 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.553713 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.553177 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp"] Oct 29 22:50:58.557546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.557513 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:50:58.557680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.557654 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:50:58.643392 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.643361 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.643392 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.643399 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.643753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.643540 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74pq\" (UniqueName: \"kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.745318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.745282 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.745490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.745330 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.745490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.745376 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74pq\" (UniqueName: \"kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.746007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.745984 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.747680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.747660 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.771044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.771019 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74pq\" (UniqueName: \"kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq\") pod \"maas-api-revoke-sub-ee831daa-bdnsp\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:58.845224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:58.845200 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:50:59.199784 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:50:59.199751 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfcc43c_ffa5_491d_9a41_69ce0c869823.slice/crio-8eabde3e7e416fcbe340fa2c984f0181572b680ef57e7e3aa80b7bc24e7dfe69 WatchSource:0}: Error finding container 8eabde3e7e416fcbe340fa2c984f0181572b680ef57e7e3aa80b7bc24e7dfe69: Status 404 returned error can't find the container with id 8eabde3e7e416fcbe340fa2c984f0181572b680ef57e7e3aa80b7bc24e7dfe69 Oct 29 22:50:59.201090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:50:59.201067 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp"] Oct 29 22:51:00.136365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:00.136333 5367 generic.go:356] "Generic (PLEG): container finished" podID="7dfcc43c-ffa5-491d-9a41-69ce0c869823" containerID="396a97a1888a9bd1156cd47dc9386a52c83e717a6857301b3843dca16fa13d39" exitCode=0 Oct 29 22:51:00.136756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:00.136409 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" event={"ID":"7dfcc43c-ffa5-491d-9a41-69ce0c869823","Type":"ContainerDied","Data":"396a97a1888a9bd1156cd47dc9386a52c83e717a6857301b3843dca16fa13d39"} Oct 29 22:51:00.136756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:00.136466 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" event={"ID":"7dfcc43c-ffa5-491d-9a41-69ce0c869823","Type":"ContainerStarted","Data":"8eabde3e7e416fcbe340fa2c984f0181572b680ef57e7e3aa80b7bc24e7dfe69"} Oct 29 22:51:01.335336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.335310 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:51:01.391753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.391686 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca\") pod \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " Oct 29 22:51:01.391912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.391878 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca\") pod \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " Oct 29 22:51:01.391977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.391964 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq\" (UniqueName: \"kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq\") pod \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\" (UID: \"7dfcc43c-ffa5-491d-9a41-69ce0c869823\") " Oct 29 22:51:01.392051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.392026 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca" pod "7dfcc43c-ffa5-491d-9a41-69ce0c869823" (UID: "7dfcc43c-ffa5-491d-9a41-69ce0c869823"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:01.392850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.392818 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:01.394082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.394054 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca" pod "7dfcc43c-ffa5-491d-9a41-69ce0c869823" (UID: "7dfcc43c-ffa5-491d-9a41-69ce0c869823"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:01.394305 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.394273 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq" pod "7dfcc43c-ffa5-491d-9a41-69ce0c869823" (UID: "7dfcc43c-ffa5-491d-9a41-69ce0c869823"). InnerVolumeSpecName "kube-api-access-m74pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:01.494769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.494727 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfcc43c-ffa5-491d-9a41-69ce0c869823-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:01.494769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:01.494766 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74pq\" (UniqueName: \"kubernetes.io/projected/7dfcc43c-ffa5-491d-9a41-69ce0c869823-kube-api-access-m74pq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:02.158867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:02.158837 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" Oct 29 22:51:02.159177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:02.158869 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp" event={"ID":"7dfcc43c-ffa5-491d-9a41-69ce0c869823","Type":"ContainerDied","Data":"8eabde3e7e416fcbe340fa2c984f0181572b680ef57e7e3aa80b7bc24e7dfe69"} Oct 29 22:51:02.159177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:02.158916 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eabde3e7e416fcbe340fa2c984f0181572b680ef57e7e3aa80b7bc24e7dfe69" Oct 29 22:51:22.332728 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:51:22.332677 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:51:28.562457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.562422 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png"] Oct 29 22:51:28.563679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.563655 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7dfcc43c-ffa5-491d-9a41-69ce0c869823" containerName="revoke-keys" Oct 29 22:51:28.563679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.563676 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfcc43c-ffa5-491d-9a41-69ce0c869823" containerName="revoke-keys" Oct 29 22:51:28.563883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.563867 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfcc43c-ffa5-491d-9a41-69ce0c869823" containerName="revoke-keys" Oct 29 22:51:28.567562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.567544 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.588172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.588144 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png"] Oct 29 22:51:28.595288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.595263 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:51:28.595288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.595281 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:51:28.664585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.664547 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tfb6\" (UniqueName: \"kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.664768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.664665 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.664823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.664795 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.766185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.766149 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.766374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.766246 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.766463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.766433 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tfb6\" (UniqueName: \"kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.766917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.766894 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.768665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.768643 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.786844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.786820 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tfb6\" (UniqueName: \"kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6\") pod \"maas-api-revoke-sub-fff2b4f1-n4png\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:28.882546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:28.882462 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:29.198352 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:51:29.198319 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca13b043_9da4_46f0_ac47_338440728a63.slice/crio-a357abd003f1c6a9db286cd206e90104b272efdf535d51c1786f9f1ec880a38e WatchSource:0}: Error finding container a357abd003f1c6a9db286cd206e90104b272efdf535d51c1786f9f1ec880a38e: Status 404 returned error can't find the container with id a357abd003f1c6a9db286cd206e90104b272efdf535d51c1786f9f1ec880a38e Oct 29 22:51:29.198670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:29.198649 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png"] Oct 29 22:51:29.510127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:29.510042 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" event={"ID":"ca13b043-9da4-46f0-ac47-338440728a63","Type":"ContainerStarted","Data":"a357abd003f1c6a9db286cd206e90104b272efdf535d51c1786f9f1ec880a38e"} Oct 29 22:51:30.522099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:30.522015 5367 generic.go:356] "Generic (PLEG): container finished" podID="ca13b043-9da4-46f0-ac47-338440728a63" containerID="8682b2e6034a937531173eb61454b2bf77634abd297f021f79a804b02efdb48b" exitCode=0 Oct 29 22:51:30.522099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:30.522070 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" event={"ID":"ca13b043-9da4-46f0-ac47-338440728a63","Type":"ContainerDied","Data":"8682b2e6034a937531173eb61454b2bf77634abd297f021f79a804b02efdb48b"} Oct 29 22:51:31.722610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.722577 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:31.819443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.819406 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6\" (UniqueName: \"kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6\") pod \"ca13b043-9da4-46f0-ac47-338440728a63\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " Oct 29 22:51:31.819649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.819497 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca\") pod \"ca13b043-9da4-46f0-ac47-338440728a63\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " Oct 29 22:51:31.819829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.819655 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca\") pod \"ca13b043-9da4-46f0-ac47-338440728a63\" (UID: \"ca13b043-9da4-46f0-ac47-338440728a63\") " Oct 29 22:51:31.820040 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.820013 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca" pod "ca13b043-9da4-46f0-ac47-338440728a63" (UID: "ca13b043-9da4-46f0-ac47-338440728a63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:31.820737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.820683 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:31.821927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.821902 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca" pod "ca13b043-9da4-46f0-ac47-338440728a63" (UID: "ca13b043-9da4-46f0-ac47-338440728a63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:31.821927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.821913 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6" pod "ca13b043-9da4-46f0-ac47-338440728a63" (UID: "ca13b043-9da4-46f0-ac47-338440728a63"). InnerVolumeSpecName "kube-api-access-9tfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:31.921998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.921965 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tfb6\" (UniqueName: \"kubernetes.io/projected/ca13b043-9da4-46f0-ac47-338440728a63-kube-api-access-9tfb6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:31.921998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:31.921994 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca13b043-9da4-46f0-ac47-338440728a63-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:32.541744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:32.541713 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" Oct 29 22:51:32.541931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:32.541732 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png" event={"ID":"ca13b043-9da4-46f0-ac47-338440728a63","Type":"ContainerDied","Data":"a357abd003f1c6a9db286cd206e90104b272efdf535d51c1786f9f1ec880a38e"} Oct 29 22:51:32.541931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:32.541794 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a357abd003f1c6a9db286cd206e90104b272efdf535d51c1786f9f1ec880a38e" Oct 29 22:51:36.246687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.246653 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f"] Oct 29 22:51:36.247522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.247505 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ca13b043-9da4-46f0-ac47-338440728a63" containerName="revoke-keys" Oct 29 22:51:36.247567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.247522 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca13b043-9da4-46f0-ac47-338440728a63" containerName="revoke-keys" Oct 29 22:51:36.247682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.247673 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca13b043-9da4-46f0-ac47-338440728a63" containerName="revoke-keys" Oct 29 22:51:36.251910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.251894 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.272492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.272462 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f"] Oct 29 22:51:36.279947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.279919 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:51:36.280080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.279967 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:51:36.314599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.314558 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwrp\" (UniqueName: \"kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.314791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.314635 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.314957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.314884 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.420082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.420041 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwrp\" (UniqueName: \"kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.421820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.421793 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.421956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.421852 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.422073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.422054 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.426387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.425720 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.442826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.442791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwrp\" (UniqueName: \"kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp\") pod \"maas-api-revoke-sub-489b1cc0-njj4f\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.495984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.495953 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj"] Oct 29 22:51:36.501404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.501329 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.523536 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.523507 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.523773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.523753 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl9rj\" (UniqueName: \"kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.523838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.523805 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.523881 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.523858 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj"] Oct 29 22:51:36.566662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.566633 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:36.625084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.625048 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9rj\" (UniqueName: \"kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.625253 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.625107 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.625848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.625710 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.625937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.625925 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.635185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.635159 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.659015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.658987 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9rj\" (UniqueName: \"kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj\") pod \"maas-api-revoke-sub-41c23957-vxdlj\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.817498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.817470 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:36.891470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:36.891449 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f"] Oct 29 22:51:37.133348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.133314 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj"] Oct 29 22:51:37.156790 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:51:37.156760 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cf1d86_f552_4e77_9fec_9a62e9535ce8.slice/crio-8087614e23933690f7e099edbde7ee20d7df82ff7ef2f2f2926f35febee66421 WatchSource:0}: Error finding container 8087614e23933690f7e099edbde7ee20d7df82ff7ef2f2f2926f35febee66421: Status 404 returned error can't find the container with id 8087614e23933690f7e099edbde7ee20d7df82ff7ef2f2f2926f35febee66421 Oct 29 22:51:37.165610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.165584 5367 scope.go:122] "RemoveContainer" containerID="51362b4af5fde51902ce1a1217b00f6f21b3e41d9a5cceba70f3636a70563396" Oct 29 22:51:37.176650 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.176629 5367 scope.go:122] "RemoveContainer" containerID="9c041f0c61642b54b01234c127eb66685063bc7566b26855f3820d66e861f502" Oct 29 22:51:37.192074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.192050 5367 scope.go:122] "RemoveContainer" containerID="f30ab670209a905ed88798f8aeac53329c94dde18a7dac03cf2bc02afc74be8f" Oct 29 22:51:37.204044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.204021 5367 scope.go:122] "RemoveContainer" containerID="c3f179365f750d532f675e48ef943485bfea01b6e4fb7b159f88f20d8d1d970c" Oct 29 22:51:37.619019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.618973 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" event={"ID":"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e","Type":"ContainerStarted","Data":"4d3053f6fa4de441e7f4a98ddaff442a5fd8e3e9914de87b3494eab7efcd5054"} Oct 29 22:51:37.619523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.619082 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" event={"ID":"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e","Type":"ContainerStarted","Data":"045d1e8e6bfde79e85440f1d96d6d10980b3c6e259c7ba933e939877259d4043"} Oct 29 22:51:37.622871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.622191 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" event={"ID":"b4cf1d86-f552-4e77-9fec-9a62e9535ce8","Type":"ContainerStarted","Data":"8087614e23933690f7e099edbde7ee20d7df82ff7ef2f2f2926f35febee66421"} Oct 29 22:51:37.665962 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:37.665911 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" podStartSLOduration=1.665897314 podStartE2EDuration="1.665897314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:37.66198642 +0000 UTC m=+2164.556061354" watchObservedRunningTime="2026-10-29 22:51:37.665897314 +0000 UTC m=+2164.559972246" Oct 29 22:51:38.632981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:38.632940 5367 generic.go:356] "Generic (PLEG): container finished" podID="05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" containerID="4d3053f6fa4de441e7f4a98ddaff442a5fd8e3e9914de87b3494eab7efcd5054" exitCode=0 Oct 29 22:51:38.633480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:38.632989 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" event={"ID":"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e","Type":"ContainerDied","Data":"4d3053f6fa4de441e7f4a98ddaff442a5fd8e3e9914de87b3494eab7efcd5054"} Oct 29 22:51:38.635258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:38.635226 5367 generic.go:356] "Generic (PLEG): container finished" podID="b4cf1d86-f552-4e77-9fec-9a62e9535ce8" containerID="139f97ef551601c0614489897e793aab6a9bbc253605d749bccc800c08505acb" exitCode=0 Oct 29 22:51:38.635370 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:38.635306 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" event={"ID":"b4cf1d86-f552-4e77-9fec-9a62e9535ce8","Type":"ContainerDied","Data":"139f97ef551601c0614489897e793aab6a9bbc253605d749bccc800c08505acb"} Oct 29 22:51:39.894531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.894501 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:39.940999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.940972 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:39.987526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.987495 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca\") pod \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " Oct 29 22:51:39.987909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.987884 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca\") pod \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " Oct 29 22:51:39.988083 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.988063 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca" pod "b4cf1d86-f552-4e77-9fec-9a62e9535ce8" (UID: "b4cf1d86-f552-4e77-9fec-9a62e9535ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:39.988221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.988198 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca\") pod \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " Oct 29 22:51:39.988418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.988305 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca\") pod \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " Oct 29 22:51:39.988418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.988374 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp\" (UniqueName: \"kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp\") pod \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\" (UID: \"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e\") " Oct 29 22:51:39.988631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.988450 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj\" (UniqueName: \"kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj\") pod \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\" (UID: \"b4cf1d86-f552-4e77-9fec-9a62e9535ce8\") " Oct 29 22:51:39.988943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.988915 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca" pod "05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" (UID: "05c1b62b-9cb0-4225-bb65-f7bb7eaf179e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:39.989229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.989208 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:39.989330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.989236 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:39.991565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.991543 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca" pod "b4cf1d86-f552-4e77-9fec-9a62e9535ce8" (UID: "b4cf1d86-f552-4e77-9fec-9a62e9535ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:39.991904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.991878 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca" pod "05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" (UID: "05c1b62b-9cb0-4225-bb65-f7bb7eaf179e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:39.992073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.992040 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj" pod "b4cf1d86-f552-4e77-9fec-9a62e9535ce8" (UID: "b4cf1d86-f552-4e77-9fec-9a62e9535ce8"). InnerVolumeSpecName "kube-api-access-tl9rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:39.992257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:39.992237 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp" pod "05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" (UID: "05c1b62b-9cb0-4225-bb65-f7bb7eaf179e"). InnerVolumeSpecName "kube-api-access-kcwrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:40.090478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.090443 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.090478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.090474 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwrp\" (UniqueName: \"kubernetes.io/projected/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-kube-api-access-kcwrp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.090478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.090486 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl9rj\" (UniqueName: \"kubernetes.io/projected/b4cf1d86-f552-4e77-9fec-9a62e9535ce8-kube-api-access-tl9rj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.090846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.090496 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.655284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.655253 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" Oct 29 22:51:40.655456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.655283 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f" event={"ID":"05c1b62b-9cb0-4225-bb65-f7bb7eaf179e","Type":"ContainerDied","Data":"045d1e8e6bfde79e85440f1d96d6d10980b3c6e259c7ba933e939877259d4043"} Oct 29 22:51:40.655456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.655333 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="045d1e8e6bfde79e85440f1d96d6d10980b3c6e259c7ba933e939877259d4043" Oct 29 22:51:40.657023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.656999 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" Oct 29 22:51:40.657128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.657026 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj" event={"ID":"b4cf1d86-f552-4e77-9fec-9a62e9535ce8","Type":"ContainerDied","Data":"8087614e23933690f7e099edbde7ee20d7df82ff7ef2f2f2926f35febee66421"} Oct 29 22:51:40.657128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:40.657058 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8087614e23933690f7e099edbde7ee20d7df82ff7ef2f2f2926f35febee66421" Oct 29 22:51:48.331490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.331405 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh"] Oct 29 22:51:48.332317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.332299 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b4cf1d86-f552-4e77-9fec-9a62e9535ce8" containerName="revoke-keys" Oct 29 22:51:48.332364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.332318 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cf1d86-f552-4e77-9fec-9a62e9535ce8" containerName="revoke-keys" Oct 29 22:51:48.332364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.332344 5367 cpu_manager.go:414] "containerMap: removing container" podUID="05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" containerName="revoke-keys" Oct 29 22:51:48.332364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.332349 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" containerName="revoke-keys" Oct 29 22:51:48.332508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.332498 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cf1d86-f552-4e77-9fec-9a62e9535ce8" containerName="revoke-keys" Oct 29 22:51:48.332545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.332513 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" containerName="revoke-keys" Oct 29 22:51:48.336892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.336870 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.342767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.342736 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.342883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.342793 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbl9v\" (UniqueName: \"kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.343242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.343221 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.352805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.352759 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:51:48.359281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.359258 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh"] Oct 29 22:51:48.365860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.365841 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:51:48.473815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.471205 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.473815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.471350 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.473815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.471402 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbl9v\" (UniqueName: \"kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.473815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.472747 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.477798 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.476086 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.503963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.503933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbl9v\" (UniqueName: \"kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v\") pod \"maas-api-revoke-sub-8c83b040-hj5fh\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.652815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.652743 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:48.970206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:48.970174 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh"] Oct 29 22:51:48.971333 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:51:48.971303 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36386cd_ba0d_43d2_a610_f0aaed46a1e8.slice/crio-72ee4baaf31ebd4b4b9ddc259bc7459834d34b9fccd146be06afc5979f2d1377 WatchSource:0}: Error finding container 72ee4baaf31ebd4b4b9ddc259bc7459834d34b9fccd146be06afc5979f2d1377: Status 404 returned error can't find the container with id 72ee4baaf31ebd4b4b9ddc259bc7459834d34b9fccd146be06afc5979f2d1377 Oct 29 22:51:49.743224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:49.743144 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" event={"ID":"d36386cd-ba0d-43d2-a610-f0aaed46a1e8","Type":"ContainerStarted","Data":"8608d6372fd976138f997bbf0dc804cb77a85dd4b36dfe38fdbe852bf8f0d828"} Oct 29 22:51:49.743224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:49.743205 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" event={"ID":"d36386cd-ba0d-43d2-a610-f0aaed46a1e8","Type":"ContainerStarted","Data":"72ee4baaf31ebd4b4b9ddc259bc7459834d34b9fccd146be06afc5979f2d1377"} Oct 29 22:51:49.788887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:49.788835 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" podStartSLOduration=1.788821394 podStartE2EDuration="1.788821394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:49.784943776 +0000 UTC m=+2176.679018697" watchObservedRunningTime="2026-10-29 22:51:49.788821394 +0000 UTC m=+2176.682896314" Oct 29 22:51:50.754567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:50.754530 5367 generic.go:356] "Generic (PLEG): container finished" podID="d36386cd-ba0d-43d2-a610-f0aaed46a1e8" containerID="8608d6372fd976138f997bbf0dc804cb77a85dd4b36dfe38fdbe852bf8f0d828" exitCode=0 Oct 29 22:51:50.755070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:50.754614 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" event={"ID":"d36386cd-ba0d-43d2-a610-f0aaed46a1e8","Type":"ContainerDied","Data":"8608d6372fd976138f997bbf0dc804cb77a85dd4b36dfe38fdbe852bf8f0d828"} Oct 29 22:51:51.962545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:51.962515 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:52.036869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.036783 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca\") pod \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " Oct 29 22:51:52.036869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.036824 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca\") pod \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " Oct 29 22:51:52.037111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.036892 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v\" (UniqueName: \"kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v\") pod \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\" (UID: \"d36386cd-ba0d-43d2-a610-f0aaed46a1e8\") " Oct 29 22:51:52.037357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.037328 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca" pod "d36386cd-ba0d-43d2-a610-f0aaed46a1e8" (UID: "d36386cd-ba0d-43d2-a610-f0aaed46a1e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:52.037729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.037710 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:52.039659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.039632 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca" pod "d36386cd-ba0d-43d2-a610-f0aaed46a1e8" (UID: "d36386cd-ba0d-43d2-a610-f0aaed46a1e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:52.039854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.039826 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v" pod "d36386cd-ba0d-43d2-a610-f0aaed46a1e8" (UID: "d36386cd-ba0d-43d2-a610-f0aaed46a1e8"). InnerVolumeSpecName "kube-api-access-rbl9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:52.139357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.139322 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:52.139357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.139350 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbl9v\" (UniqueName: \"kubernetes.io/projected/d36386cd-ba0d-43d2-a610-f0aaed46a1e8-kube-api-access-rbl9v\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:52.333584 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:51:52.333545 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:51:52.786116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.786034 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" event={"ID":"d36386cd-ba0d-43d2-a610-f0aaed46a1e8","Type":"ContainerDied","Data":"72ee4baaf31ebd4b4b9ddc259bc7459834d34b9fccd146be06afc5979f2d1377"} Oct 29 22:51:52.786116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.786073 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh" Oct 29 22:51:52.786116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.786110 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72ee4baaf31ebd4b4b9ddc259bc7459834d34b9fccd146be06afc5979f2d1377" Oct 29 22:51:52.831959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.831924 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6"] Oct 29 22:51:52.833420 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.833387 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d36386cd-ba0d-43d2-a610-f0aaed46a1e8" containerName="revoke-keys" Oct 29 22:51:52.833420 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.833416 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36386cd-ba0d-43d2-a610-f0aaed46a1e8" containerName="revoke-keys" Oct 29 22:51:52.833655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.833638 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36386cd-ba0d-43d2-a610-f0aaed46a1e8" containerName="revoke-keys" Oct 29 22:51:52.838750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.838728 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:52.854310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.854266 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:51:52.854544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.854399 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6"] Oct 29 22:51:52.854544 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.854438 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:51:52.957655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.957626 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gxm\" (UniqueName: \"kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:52.957844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.957668 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:52.957984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:52.957960 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.059838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.059810 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gxm\" (UniqueName: \"kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.060231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.059855 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.060231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.060023 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.060626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.060605 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.062270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.062243 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.081175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.081152 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gxm\" (UniqueName: \"kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm\") pod \"maas-api-revoke-sub-6cefa3b9-4d5s6\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.158354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.158325 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:53.473614 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:51:53.473577 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba88738_149e_4268_a5e9_b354769d568f.slice/crio-4679419f48cd1381250cd236f8b753993127b645ec5458fbf795b641f24310c1 WatchSource:0}: Error finding container 4679419f48cd1381250cd236f8b753993127b645ec5458fbf795b641f24310c1: Status 404 returned error can't find the container with id 4679419f48cd1381250cd236f8b753993127b645ec5458fbf795b641f24310c1 Oct 29 22:51:53.474367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.474343 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6"] Oct 29 22:51:53.796959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:53.796918 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" event={"ID":"fba88738-149e-4268-a5e9-b354769d568f","Type":"ContainerStarted","Data":"4679419f48cd1381250cd236f8b753993127b645ec5458fbf795b641f24310c1"} Oct 29 22:51:54.808089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:54.808053 5367 generic.go:356] "Generic (PLEG): container finished" podID="fba88738-149e-4268-a5e9-b354769d568f" containerID="7aa5e59a9f007ada6fe83e197b589fbcdbd32e85dca77f3f8194728fe5bbeade" exitCode=0 Oct 29 22:51:54.808489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:54.808134 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" event={"ID":"fba88738-149e-4268-a5e9-b354769d568f","Type":"ContainerDied","Data":"7aa5e59a9f007ada6fe83e197b589fbcdbd32e85dca77f3f8194728fe5bbeade"} Oct 29 22:51:56.012248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.012225 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:56.066795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.066767 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm\" (UniqueName: \"kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm\") pod \"fba88738-149e-4268-a5e9-b354769d568f\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " Oct 29 22:51:56.066943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.066930 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca\") pod \"fba88738-149e-4268-a5e9-b354769d568f\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " Oct 29 22:51:56.067138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.067121 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca\") pod \"fba88738-149e-4268-a5e9-b354769d568f\" (UID: \"fba88738-149e-4268-a5e9-b354769d568f\") " Oct 29 22:51:56.067351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.067326 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca" pod "fba88738-149e-4268-a5e9-b354769d568f" (UID: "fba88738-149e-4268-a5e9-b354769d568f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:56.067708 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.067668 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba88738-149e-4268-a5e9-b354769d568f-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:56.069006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.068981 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm" pod "fba88738-149e-4268-a5e9-b354769d568f" (UID: "fba88738-149e-4268-a5e9-b354769d568f"). InnerVolumeSpecName "kube-api-access-g4gxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:56.069153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.069120 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca" pod "fba88738-149e-4268-a5e9-b354769d568f" (UID: "fba88738-149e-4268-a5e9-b354769d568f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:56.169400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.169365 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba88738-149e-4268-a5e9-b354769d568f-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:56.169400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.169395 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gxm\" (UniqueName: \"kubernetes.io/projected/fba88738-149e-4268-a5e9-b354769d568f-kube-api-access-g4gxm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:51:56.841394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.841362 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" Oct 29 22:51:56.841394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.841385 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6" event={"ID":"fba88738-149e-4268-a5e9-b354769d568f","Type":"ContainerDied","Data":"4679419f48cd1381250cd236f8b753993127b645ec5458fbf795b641f24310c1"} Oct 29 22:51:56.841609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:51:56.841431 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4679419f48cd1381250cd236f8b753993127b645ec5458fbf795b641f24310c1" Oct 29 22:52:02.678499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.678450 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2"] Oct 29 22:52:02.699725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.694720 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fba88738-149e-4268-a5e9-b354769d568f" containerName="revoke-keys" Oct 29 22:52:02.700021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.699996 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba88738-149e-4268-a5e9-b354769d568f" containerName="revoke-keys" Oct 29 22:52:02.711795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.705876 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba88738-149e-4268-a5e9-b354769d568f" containerName="revoke-keys" Oct 29 22:52:02.715801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.715770 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2"] Oct 29 22:52:02.715932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.715892 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.732023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.731938 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:52:02.745720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.745647 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:52:02.812690 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.812654 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.812913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.812737 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kff\" (UniqueName: \"kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.812983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.812964 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.903298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.903265 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb"] Oct 29 22:52:02.914586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.914541 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.914768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.914625 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kff\" (UniqueName: \"kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.914928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.914905 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.915390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.915364 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.917241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.917221 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:02.935566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:02.935490 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kff\" (UniqueName: \"kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff\") pod \"maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:03.052069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.052034 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:03.204371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.203085 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8"] Oct 29 22:52:03.209448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.208980 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.215283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.213533 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8"] Oct 29 22:52:03.339406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.339371 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.339578 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.339531 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.339850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.339833 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhnk\" (UniqueName: \"kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.391748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.391710 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2"] Oct 29 22:52:03.399476 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:52:03.399445 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e79fa3c_9808_4d39_84a9_609a57c44e7d.slice/crio-f93287d16d8080ef29854b93b8f81a709da37d0db66a7ba841b31e84343caf4d WatchSource:0}: Error finding container f93287d16d8080ef29854b93b8f81a709da37d0db66a7ba841b31e84343caf4d: Status 404 returned error can't find the container with id f93287d16d8080ef29854b93b8f81a709da37d0db66a7ba841b31e84343caf4d Oct 29 22:52:03.441971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.441940 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.442220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.442198 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhnk\" (UniqueName: \"kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.442396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.442370 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.443049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.443025 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.444477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.444456 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.463187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.463136 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhnk\" (UniqueName: \"kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk\") pod \"maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.543042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.543010 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:03.857431 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:52:03.857388 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd60df13_92f3_47f8_8c6b_68571c94a88c.slice/crio-2dfebdba93ac5877a90610298a8292d78ebfb99bc481cd3ebd8a08f89d3b8014 WatchSource:0}: Error finding container 2dfebdba93ac5877a90610298a8292d78ebfb99bc481cd3ebd8a08f89d3b8014: Status 404 returned error can't find the container with id 2dfebdba93ac5877a90610298a8292d78ebfb99bc481cd3ebd8a08f89d3b8014 Oct 29 22:52:03.857916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.857474 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8"] Oct 29 22:52:03.917042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.917007 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" event={"ID":"2e79fa3c-9808-4d39-84a9-609a57c44e7d","Type":"ContainerStarted","Data":"113439ea84f9d18703bbe6e4766eb04b308a241b4aac8b542c8658ef0d038ede"} Oct 29 22:52:03.917162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.917070 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" event={"ID":"2e79fa3c-9808-4d39-84a9-609a57c44e7d","Type":"ContainerStarted","Data":"f93287d16d8080ef29854b93b8f81a709da37d0db66a7ba841b31e84343caf4d"} Oct 29 22:52:03.918757 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.918736 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" event={"ID":"bd60df13-92f3-47f8-8c6b-68571c94a88c","Type":"ContainerStarted","Data":"2dfebdba93ac5877a90610298a8292d78ebfb99bc481cd3ebd8a08f89d3b8014"} Oct 29 22:52:03.966281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:03.966219 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" podStartSLOduration=1.966201707 podStartE2EDuration="1.966201707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:03.958957188 +0000 UTC m=+2190.853032111" watchObservedRunningTime="2026-10-29 22:52:03.966201707 +0000 UTC m=+2190.860276630" Oct 29 22:52:04.931512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:04.931461 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" event={"ID":"bd60df13-92f3-47f8-8c6b-68571c94a88c","Type":"ContainerStarted","Data":"ad4e5cf196eb65f9653534de5379599092d6f50808e3c66e61c9b578e3516069"} Oct 29 22:52:04.934656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:04.934628 5367 generic.go:356] "Generic (PLEG): container finished" podID="2e79fa3c-9808-4d39-84a9-609a57c44e7d" containerID="113439ea84f9d18703bbe6e4766eb04b308a241b4aac8b542c8658ef0d038ede" exitCode=0 Oct 29 22:52:04.934833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:04.934716 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" event={"ID":"2e79fa3c-9808-4d39-84a9-609a57c44e7d","Type":"ContainerDied","Data":"113439ea84f9d18703bbe6e4766eb04b308a241b4aac8b542c8658ef0d038ede"} Oct 29 22:52:04.979284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:04.979229 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" podStartSLOduration=1.9792106999999999 podStartE2EDuration="1.9792107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:04.973507864 +0000 UTC m=+2191.867582776" watchObservedRunningTime="2026-10-29 22:52:04.9792107 +0000 UTC m=+2191.873285623" Oct 29 22:52:06.182725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:06.181024 5367 generic.go:356] "Generic (PLEG): container finished" podID="bd60df13-92f3-47f8-8c6b-68571c94a88c" containerID="ad4e5cf196eb65f9653534de5379599092d6f50808e3c66e61c9b578e3516069" exitCode=0 Oct 29 22:52:06.209765 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:06.204009 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" event={"ID":"bd60df13-92f3-47f8-8c6b-68571c94a88c","Type":"ContainerDied","Data":"ad4e5cf196eb65f9653534de5379599092d6f50808e3c66e61c9b578e3516069"} Oct 29 22:52:07.174770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.174740 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:07.227014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.226983 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" Oct 29 22:52:07.233911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.229393 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2" event={"ID":"2e79fa3c-9808-4d39-84a9-609a57c44e7d","Type":"ContainerDied","Data":"f93287d16d8080ef29854b93b8f81a709da37d0db66a7ba841b31e84343caf4d"} Oct 29 22:52:07.234142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.234125 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93287d16d8080ef29854b93b8f81a709da37d0db66a7ba841b31e84343caf4d" Oct 29 22:52:07.290086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.289276 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca\") pod \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " Oct 29 22:52:07.290086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.289478 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff\" (UniqueName: \"kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff\") pod \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " Oct 29 22:52:07.290086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.289518 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca\") pod \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\" (UID: \"2e79fa3c-9808-4d39-84a9-609a57c44e7d\") " Oct 29 22:52:07.308560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.308526 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca" pod "2e79fa3c-9808-4d39-84a9-609a57c44e7d" (UID: "2e79fa3c-9808-4d39-84a9-609a57c44e7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:07.331078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.331027 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca" pod "2e79fa3c-9808-4d39-84a9-609a57c44e7d" (UID: "2e79fa3c-9808-4d39-84a9-609a57c44e7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:07.333467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.333424 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff" pod "2e79fa3c-9808-4d39-84a9-609a57c44e7d" (UID: "2e79fa3c-9808-4d39-84a9-609a57c44e7d"). InnerVolumeSpecName "kube-api-access-m6kff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:07.422722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.417491 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6kff\" (UniqueName: \"kubernetes.io/projected/2e79fa3c-9808-4d39-84a9-609a57c44e7d-kube-api-access-m6kff\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.422722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.417531 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.422722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.417547 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e79fa3c-9808-4d39-84a9-609a57c44e7d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.854738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.854672 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:07.934058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.933971 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk\" (UniqueName: \"kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk\") pod \"bd60df13-92f3-47f8-8c6b-68571c94a88c\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " Oct 29 22:52:07.934237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.934063 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca\") pod \"bd60df13-92f3-47f8-8c6b-68571c94a88c\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " Oct 29 22:52:07.934237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.934092 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca\") pod \"bd60df13-92f3-47f8-8c6b-68571c94a88c\" (UID: \"bd60df13-92f3-47f8-8c6b-68571c94a88c\") " Oct 29 22:52:07.934622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.934588 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca" pod "bd60df13-92f3-47f8-8c6b-68571c94a88c" (UID: "bd60df13-92f3-47f8-8c6b-68571c94a88c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:07.935053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.935027 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.936336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.936313 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca" pod "bd60df13-92f3-47f8-8c6b-68571c94a88c" (UID: "bd60df13-92f3-47f8-8c6b-68571c94a88c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:07.936516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:07.936491 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk" pod "bd60df13-92f3-47f8-8c6b-68571c94a88c" (UID: "bd60df13-92f3-47f8-8c6b-68571c94a88c"). InnerVolumeSpecName "kube-api-access-qvhnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:08.036879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:08.036845 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhnk\" (UniqueName: \"kubernetes.io/projected/bd60df13-92f3-47f8-8c6b-68571c94a88c-kube-api-access-qvhnk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:08.036879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:08.036871 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd60df13-92f3-47f8-8c6b-68571c94a88c-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:08.237022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:08.236757 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" Oct 29 22:52:08.237407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:08.237196 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8" event={"ID":"bd60df13-92f3-47f8-8c6b-68571c94a88c","Type":"ContainerDied","Data":"2dfebdba93ac5877a90610298a8292d78ebfb99bc481cd3ebd8a08f89d3b8014"} Oct 29 22:52:08.237407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:08.237253 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dfebdba93ac5877a90610298a8292d78ebfb99bc481cd3ebd8a08f89d3b8014" Oct 29 22:52:13.023746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:13.023672 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt"] Oct 29 22:52:13.057936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:13.057898 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74"] Oct 29 22:52:13.792950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:13.792904 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl"] Oct 29 22:52:13.819839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:13.819676 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw"] Oct 29 22:52:17.926504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:17.926461 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" podUID="3b64379c-50a1-4938-ac56-5a49ab326919" containerName="main" containerID="cri-o://eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a" gracePeriod=45 Oct 29 22:52:18.024286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.024247 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerName="payload-processing" containerID="cri-o://9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b" gracePeriod=25 Oct 29 22:52:18.059006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.058969 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerName="payload-pre-processing" containerID="cri-o://4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9" gracePeriod=25 Oct 29 22:52:18.225489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.225465 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:52:18.287597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.287555 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs\") pod \"3b64379c-50a1-4938-ac56-5a49ab326919\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " Oct 29 22:52:18.287841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.287627 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home\") pod \"3b64379c-50a1-4938-ac56-5a49ab326919\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " Oct 29 22:52:18.287841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.287649 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm\") pod \"3b64379c-50a1-4938-ac56-5a49ab326919\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " Oct 29 22:52:18.287841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.287745 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir\") pod \"3b64379c-50a1-4938-ac56-5a49ab326919\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " Oct 29 22:52:18.288035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.287847 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw\" (UniqueName: \"kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw\") pod \"3b64379c-50a1-4938-ac56-5a49ab326919\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " Oct 29 22:52:18.290173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.288100 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir" pod "3b64379c-50a1-4938-ac56-5a49ab326919" (UID: "3b64379c-50a1-4938-ac56-5a49ab326919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:18.290173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.288226 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache\") pod \"3b64379c-50a1-4938-ac56-5a49ab326919\" (UID: \"3b64379c-50a1-4938-ac56-5a49ab326919\") " Oct 29 22:52:18.290173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.288495 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache" pod "3b64379c-50a1-4938-ac56-5a49ab326919" (UID: "3b64379c-50a1-4938-ac56-5a49ab326919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:18.290173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.289261 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.290173 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.289286 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.290671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.290640 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home" pod "3b64379c-50a1-4938-ac56-5a49ab326919" (UID: "3b64379c-50a1-4938-ac56-5a49ab326919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:18.292078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.292053 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs" pod "3b64379c-50a1-4938-ac56-5a49ab326919" (UID: "3b64379c-50a1-4938-ac56-5a49ab326919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:18.292182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.292116 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm" pod "3b64379c-50a1-4938-ac56-5a49ab326919" (UID: "3b64379c-50a1-4938-ac56-5a49ab326919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:18.292241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.292196 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw" pod "3b64379c-50a1-4938-ac56-5a49ab326919" (UID: "3b64379c-50a1-4938-ac56-5a49ab326919"). InnerVolumeSpecName "kube-api-access-5k6gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:18.390990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.390961 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b64379c-50a1-4938-ac56-5a49ab326919-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.390990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.390988 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.390990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.390996 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b64379c-50a1-4938-ac56-5a49ab326919-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.391214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.391005 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6gw\" (UniqueName: \"kubernetes.io/projected/3b64379c-50a1-4938-ac56-5a49ab326919-kube-api-access-5k6gw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.403020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.402988 5367 generic.go:356] "Generic (PLEG): container finished" podID="3b64379c-50a1-4938-ac56-5a49ab326919" containerID="eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a" exitCode=0 Oct 29 22:52:18.403170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.403027 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" event={"ID":"3b64379c-50a1-4938-ac56-5a49ab326919","Type":"ContainerDied","Data":"eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a"} Oct 29 22:52:18.403170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.403032 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" Oct 29 22:52:18.403170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.403060 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb" event={"ID":"3b64379c-50a1-4938-ac56-5a49ab326919","Type":"ContainerDied","Data":"b0b607ad84915aba7e20cf954fcd0d2d6c3ef272b2fe77486f51a0030de18044"} Oct 29 22:52:18.403170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.403078 5367 scope.go:122] "RemoveContainer" containerID="eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a" Oct 29 22:52:18.415628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.415606 5367 scope.go:122] "RemoveContainer" containerID="eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a" Oct 29 22:52:18.415928 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:18.415908 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a\": container with ID starting with eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a not found: ID does not exist" containerID="eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a" Oct 29 22:52:18.415997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.415940 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a"} err="failed to get container status \"eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a\": rpc error: code = NotFound desc = could not find container \"eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a\": container with ID starting with eaa07d9718ca2e6e853aa91c7abd072ea594048a32985ce90e66a215e34df92a not found: ID does not exist" Oct 29 22:52:18.461031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.460971 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb"] Oct 29 22:52:18.466251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.466229 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ad69b4/test-model-ad69b4-kserve-8544764ff5-vrfgb"] Oct 29 22:52:18.793886 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.793798 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerName="payload-processing" containerID="cri-o://5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496" gracePeriod=25 Oct 29 22:52:18.820262 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:18.820228 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerName="payload-pre-processing" containerID="cri-o://3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e" gracePeriod=25 Oct 29 22:52:19.227005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:19.226966 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b64379c-50a1-4938-ac56-5a49ab326919" path="/var/lib/kubelet/pods/3b64379c-50a1-4938-ac56-5a49ab326919/volumes" Oct 29 22:52:19.369153 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:19.369120 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:21.491001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:21.490957 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:21.564833 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:21.564796 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:21.565012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:21.564856 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:22.382227 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:22.382193 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:52:22.533922 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:22.533887 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8"] Oct 29 22:52:22.545086 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:22.545051 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-379a6d-bqsj8"] Oct 29 22:52:23.225688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:23.225656 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd60df13-92f3-47f8-8c6b-68571c94a88c" path="/var/lib/kubelet/pods/bd60df13-92f3-47f8-8c6b-68571c94a88c/volumes" Oct 29 22:52:29.392162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:29.392117 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:30.564449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.564414 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9"] Oct 29 22:52:30.565768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.565742 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2e79fa3c-9808-4d39-84a9-609a57c44e7d" containerName="revoke-keys" Oct 29 22:52:30.565836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.565771 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e79fa3c-9808-4d39-84a9-609a57c44e7d" containerName="revoke-keys" Oct 29 22:52:30.565836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.565790 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bd60df13-92f3-47f8-8c6b-68571c94a88c" containerName="revoke-keys" Oct 29 22:52:30.565836 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.565798 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd60df13-92f3-47f8-8c6b-68571c94a88c" containerName="revoke-keys" Oct 29 22:52:30.565937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.565839 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3b64379c-50a1-4938-ac56-5a49ab326919" containerName="main" Oct 29 22:52:30.565937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.565847 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b64379c-50a1-4938-ac56-5a49ab326919" containerName="main" Oct 29 22:52:30.566046 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.566032 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e79fa3c-9808-4d39-84a9-609a57c44e7d" containerName="revoke-keys" Oct 29 22:52:30.566087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.566053 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd60df13-92f3-47f8-8c6b-68571c94a88c" containerName="revoke-keys" Oct 29 22:52:30.566087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.566073 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b64379c-50a1-4938-ac56-5a49ab326919" containerName="main" Oct 29 22:52:30.569928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.569910 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.593990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.593960 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9"] Oct 29 22:52:30.599651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.599623 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:52:30.599851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.599667 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:52:30.686949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.686913 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.687140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.687093 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9kd\" (UniqueName: \"kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.687140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.687128 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.788265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.788229 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9kd\" (UniqueName: \"kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.788427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.788284 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.788491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.788424 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.789165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.789145 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.790947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.790923 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.809363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.809340 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9kd\" (UniqueName: \"kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd\") pod \"maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:30.884608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:30.884536 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:31.199168 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:52:31.199131 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f15f795_ffcc_4b85_8bbe_79491102b6cb.slice/crio-3733d7e427b707daa13373bdeb81ab7219657ec20327d904ec9a55fbfaf34835 WatchSource:0}: Error finding container 3733d7e427b707daa13373bdeb81ab7219657ec20327d904ec9a55fbfaf34835: Status 404 returned error can't find the container with id 3733d7e427b707daa13373bdeb81ab7219657ec20327d904ec9a55fbfaf34835 Oct 29 22:52:31.201582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.201414 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9"] Oct 29 22:52:31.491740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.491665 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:31.564723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.562076 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" event={"ID":"2f15f795-ffcc-4b85-8bbe-79491102b6cb","Type":"ContainerStarted","Data":"3733d7e427b707daa13373bdeb81ab7219657ec20327d904ec9a55fbfaf34835"} Oct 29 22:52:31.571037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.571003 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:31.571568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.571543 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:31.795013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.794983 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs"] Oct 29 22:52:31.800205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.800180 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:31.828045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.828015 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs"] Oct 29 22:52:31.921983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.921946 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:31.922158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.921994 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:31.922158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:31.922141 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbzl\" (UniqueName: \"kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.024651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.024534 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbzl\" (UniqueName: \"kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.024929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.024907 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.024990 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.024959 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.025693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.025670 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.027442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.027420 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.058559 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.058529 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbzl\" (UniqueName: \"kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl\") pod \"maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.127992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.127954 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:32.453363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.453333 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs"] Oct 29 22:52:32.453897 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:52:32.453874 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fd9397_0ec3_4fb8_9c5c_55c31fe967a1.slice/crio-18fada7999647ba82fe82f4b8d24c58061588ffaf005827efc12f62312d0ea17 WatchSource:0}: Error finding container 18fada7999647ba82fe82f4b8d24c58061588ffaf005827efc12f62312d0ea17: Status 404 returned error can't find the container with id 18fada7999647ba82fe82f4b8d24c58061588ffaf005827efc12f62312d0ea17 Oct 29 22:52:32.574849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.574810 5367 generic.go:356] "Generic (PLEG): container finished" podID="2f15f795-ffcc-4b85-8bbe-79491102b6cb" containerID="92cf6ecd67c1affc316b7b98a019332a214e9b87440c3949d0ac7a970c7d09c3" exitCode=0 Oct 29 22:52:32.575273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.574868 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" event={"ID":"2f15f795-ffcc-4b85-8bbe-79491102b6cb","Type":"ContainerDied","Data":"92cf6ecd67c1affc316b7b98a019332a214e9b87440c3949d0ac7a970c7d09c3"} Oct 29 22:52:32.576592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:32.576567 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" event={"ID":"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1","Type":"ContainerStarted","Data":"18fada7999647ba82fe82f4b8d24c58061588ffaf005827efc12f62312d0ea17"} Oct 29 22:52:33.658274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:33.658234 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" event={"ID":"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1","Type":"ContainerStarted","Data":"fd5ec22391c405b22eff8cc47b97f33c3cf155c0f6e79ed524c0e0f7cd79659c"} Oct 29 22:52:33.716858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:33.716795 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" podStartSLOduration=2.716776558 podStartE2EDuration="2.716776558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:33.714897842 +0000 UTC m=+2220.608972787" watchObservedRunningTime="2026-10-29 22:52:33.716776558 +0000 UTC m=+2220.610851480" Oct 29 22:52:34.572285 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.571890 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:34.670721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.667685 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd\" (UniqueName: \"kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd\") pod \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " Oct 29 22:52:34.670721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.667865 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca\") pod \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " Oct 29 22:52:34.670721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.667978 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca\") pod \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\" (UID: \"2f15f795-ffcc-4b85-8bbe-79491102b6cb\") " Oct 29 22:52:34.780338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.676475 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca" pod "2f15f795-ffcc-4b85-8bbe-79491102b6cb" (UID: "2f15f795-ffcc-4b85-8bbe-79491102b6cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:34.780338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.760988 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca" pod "2f15f795-ffcc-4b85-8bbe-79491102b6cb" (UID: "2f15f795-ffcc-4b85-8bbe-79491102b6cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:34.783475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.783400 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:34.783475 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.783449 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15f795-ffcc-4b85-8bbe-79491102b6cb-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:34.833732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.828684 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd" pod "2f15f795-ffcc-4b85-8bbe-79491102b6cb" (UID: "2f15f795-ffcc-4b85-8bbe-79491102b6cb"). InnerVolumeSpecName "kube-api-access-mm9kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:34.840579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.836562 5367 generic.go:356] "Generic (PLEG): container finished" podID="77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" containerID="fd5ec22391c405b22eff8cc47b97f33c3cf155c0f6e79ed524c0e0f7cd79659c" exitCode=0 Oct 29 22:52:34.840579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.836910 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" event={"ID":"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1","Type":"ContainerDied","Data":"fd5ec22391c405b22eff8cc47b97f33c3cf155c0f6e79ed524c0e0f7cd79659c"} Oct 29 22:52:34.840579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.839204 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" event={"ID":"2f15f795-ffcc-4b85-8bbe-79491102b6cb","Type":"ContainerDied","Data":"3733d7e427b707daa13373bdeb81ab7219657ec20327d904ec9a55fbfaf34835"} Oct 29 22:52:34.840579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.839245 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3733d7e427b707daa13373bdeb81ab7219657ec20327d904ec9a55fbfaf34835" Oct 29 22:52:34.840579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.839300 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9" Oct 29 22:52:34.887723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:34.887185 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9kd\" (UniqueName: \"kubernetes.io/projected/2f15f795-ffcc-4b85-8bbe-79491102b6cb-kube-api-access-mm9kd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:36.030198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.030168 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:36.113280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.113187 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca\") pod \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " Oct 29 22:52:36.113457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.113341 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca\") pod \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " Oct 29 22:52:36.113457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.113383 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl\" (UniqueName: \"kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl\") pod \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\" (UID: \"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1\") " Oct 29 22:52:36.113612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.113579 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca" pod "77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" (UID: "77fd9397-0ec3-4fb8-9c5c-55c31fe967a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:36.114259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.114234 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:36.115613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.115584 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca" pod "77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" (UID: "77fd9397-0ec3-4fb8-9c5c-55c31fe967a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:36.116206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.116178 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl" pod "77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" (UID: "77fd9397-0ec3-4fb8-9c5c-55c31fe967a1"). InnerVolumeSpecName "kube-api-access-9pbzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:36.215457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.215424 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:36.215457 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.215453 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbzl\" (UniqueName: \"kubernetes.io/projected/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1-kube-api-access-9pbzl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:36.860184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.860150 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" Oct 29 22:52:36.860372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.860174 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs" event={"ID":"77fd9397-0ec3-4fb8-9c5c-55c31fe967a1","Type":"ContainerDied","Data":"18fada7999647ba82fe82f4b8d24c58061588ffaf005827efc12f62312d0ea17"} Oct 29 22:52:36.860372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:36.860236 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fada7999647ba82fe82f4b8d24c58061588ffaf005827efc12f62312d0ea17" Oct 29 22:52:37.255686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:37.255608 5367 scope.go:122] "RemoveContainer" containerID="f7a62dded2682b2675d9c410e807c9916dbad17bb2750b9ff209bf50ce274a7f" Oct 29 22:52:38.364197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.364169 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:52:38.408360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.408334 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:52:38.456489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.455824 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7\" (UniqueName: \"kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7\") pod \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " Oct 29 22:52:38.456489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.455970 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume\") pod \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\" (UID: \"fae5e8d4-de0b-4ecb-ae78-7495e9586400\") " Oct 29 22:52:38.456771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.456654 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume" pod "fae5e8d4-de0b-4ecb-ae78-7495e9586400" (UID: "fae5e8d4-de0b-4ecb-ae78-7495e9586400"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:38.457118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.457093 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae5e8d4-de0b-4ecb-ae78-7495e9586400-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:38.459327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.459293 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7" pod "fae5e8d4-de0b-4ecb-ae78-7495e9586400" (UID: "fae5e8d4-de0b-4ecb-ae78-7495e9586400"). InnerVolumeSpecName "kube-api-access-vcqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:38.557708 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.557665 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume\") pod \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " Oct 29 22:52:38.558059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.558034 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume" pod "4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" (UID: "4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:38.558059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.558051 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q\" (UniqueName: \"kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q\") pod \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\" (UID: \"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9\") " Oct 29 22:52:38.558824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.558803 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcqg7\" (UniqueName: \"kubernetes.io/projected/fae5e8d4-de0b-4ecb-ae78-7495e9586400-kube-api-access-vcqg7\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:38.558824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.558826 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:38.560201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.560182 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q" pod "4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" (UID: "4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9"). InnerVolumeSpecName "kube-api-access-npd8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:38.660409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.660379 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npd8q\" (UniqueName: \"kubernetes.io/projected/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9-kube-api-access-npd8q\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:38.880832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.880800 5367 generic.go:356] "Generic (PLEG): container finished" podID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerID="4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9" exitCode=0 Oct 29 22:52:38.880994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.880851 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" Oct 29 22:52:38.880994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.880881 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" event={"ID":"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9","Type":"ContainerDied","Data":"4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9"} Oct 29 22:52:38.880994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.880934 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74" event={"ID":"4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9","Type":"ContainerDied","Data":"c6e7f0765fff418a4f2f611f23fab9594955e76097474f4816c8ae2519a65464"} Oct 29 22:52:38.880994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.880958 5367 scope.go:122] "RemoveContainer" containerID="4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9" Oct 29 22:52:38.882753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.882732 5367 generic.go:356] "Generic (PLEG): container finished" podID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerID="9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b" exitCode=0 Oct 29 22:52:38.882895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.882804 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" Oct 29 22:52:38.882895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.882841 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" event={"ID":"fae5e8d4-de0b-4ecb-ae78-7495e9586400","Type":"ContainerDied","Data":"9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b"} Oct 29 22:52:38.882895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.882874 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt" event={"ID":"fae5e8d4-de0b-4ecb-ae78-7495e9586400","Type":"ContainerDied","Data":"356cef412047c0b5ac243501f44934164358e4df895674b286bee3c291dddc5e"} Oct 29 22:52:38.937633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.937608 5367 scope.go:122] "RemoveContainer" containerID="4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9" Oct 29 22:52:38.938022 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:38.937991 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9\": container with ID starting with 4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9 not found: ID does not exist" containerID="4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9" Oct 29 22:52:38.938110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.938035 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9"} err="failed to get container status \"4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9\": rpc error: code = NotFound desc = could not find container \"4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9\": container with ID starting with 4d00999cf503062d2efbcee123e06ad8099d712a3b5c68283b28246e83c8f8e9 not found: ID does not exist" Oct 29 22:52:38.938110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.938059 5367 scope.go:122] "RemoveContainer" containerID="9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b" Oct 29 22:52:38.960337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.960304 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74"] Oct 29 22:52:38.960870 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.960849 5367 scope.go:122] "RemoveContainer" containerID="9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b" Oct 29 22:52:38.966149 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:38.965359 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b\": container with ID starting with 9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b not found: ID does not exist" containerID="9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b" Oct 29 22:52:38.966149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.965398 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b"} err="failed to get container status \"9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b\": rpc error: code = NotFound desc = could not find container \"9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b\": container with ID starting with 9c7c145e3b4a4fdd4f6ae68d06403cbc3d683155cb994b30e4cc6a7305c5de6b not found: ID does not exist" Oct 29 22:52:38.972743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:38.972613 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6bc7b2-689ff5c755-rms74"] Oct 29 22:52:39.077847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.077344 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt"] Oct 29 22:52:39.099294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.099233 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6bc7b2-7cdc874c7b-jr2xt"] Oct 29 22:52:39.180742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.180710 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:52:39.230896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.230864 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" path="/var/lib/kubelet/pods/4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9/volumes" Oct 29 22:52:39.231668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.231646 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" path="/var/lib/kubelet/pods/fae5e8d4-de0b-4ecb-ae78-7495e9586400/volumes" Oct 29 22:52:39.257505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.257478 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:52:39.273443 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.273421 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume\") pod \"a4affabc-bf17-4f8e-a803-5a73d5485724\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " Oct 29 22:52:39.273661 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.273639 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2\" (UniqueName: \"kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2\") pod \"a4affabc-bf17-4f8e-a803-5a73d5485724\" (UID: \"a4affabc-bf17-4f8e-a803-5a73d5485724\") " Oct 29 22:52:39.273957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.273931 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume" pod "a4affabc-bf17-4f8e-a803-5a73d5485724" (UID: "a4affabc-bf17-4f8e-a803-5a73d5485724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:39.274385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.274367 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4affabc-bf17-4f8e-a803-5a73d5485724-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.276449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.276423 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2" pod "a4affabc-bf17-4f8e-a803-5a73d5485724" (UID: "a4affabc-bf17-4f8e-a803-5a73d5485724"). InnerVolumeSpecName "kube-api-access-p5qk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:39.375393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.375317 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume\") pod \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " Oct 29 22:52:39.375909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.375433 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5\" (UniqueName: \"kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5\") pod \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\" (UID: \"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a\") " Oct 29 22:52:39.375909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.375745 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume" pod "7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" (UID: "7afc10ac-44a2-4b54-9f8d-f8111c94cc3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:39.376517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.376496 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qk2\" (UniqueName: \"kubernetes.io/projected/a4affabc-bf17-4f8e-a803-5a73d5485724-kube-api-access-p5qk2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.376560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.376523 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.377605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.377581 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5" pod "7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" (UID: "7afc10ac-44a2-4b54-9f8d-f8111c94cc3a"). InnerVolumeSpecName "kube-api-access-dldk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:39.478659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.478627 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldk5\" (UniqueName: \"kubernetes.io/projected/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a-kube-api-access-dldk5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.893367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.893334 5367 generic.go:356] "Generic (PLEG): container finished" podID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerID="5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496" exitCode=0 Oct 29 22:52:39.893561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.893377 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" Oct 29 22:52:39.893561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.893413 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" event={"ID":"a4affabc-bf17-4f8e-a803-5a73d5485724","Type":"ContainerDied","Data":"5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496"} Oct 29 22:52:39.893561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.893464 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl" event={"ID":"a4affabc-bf17-4f8e-a803-5a73d5485724","Type":"ContainerDied","Data":"c75794eb05c27ae159ae66f42332d3727b768cd092a9d7b1a6976c49f8b71220"} Oct 29 22:52:39.893561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.893484 5367 scope.go:122] "RemoveContainer" containerID="5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496" Oct 29 22:52:39.898100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.898077 5367 generic.go:356] "Generic (PLEG): container finished" podID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerID="3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e" exitCode=0 Oct 29 22:52:39.898206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.898114 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" Oct 29 22:52:39.898206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.898120 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" event={"ID":"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a","Type":"ContainerDied","Data":"3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e"} Oct 29 22:52:39.898206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.898153 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw" event={"ID":"7afc10ac-44a2-4b54-9f8d-f8111c94cc3a","Type":"ContainerDied","Data":"f1f3e920c7c4b17d9cc8e31159f4b1a76edd41aaa244cd9ca7ae47c98386fde4"} Oct 29 22:52:39.907623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.907540 5367 scope.go:122] "RemoveContainer" containerID="5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496" Oct 29 22:52:39.907873 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:39.907846 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496\": container with ID starting with 5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496 not found: ID does not exist" containerID="5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496" Oct 29 22:52:39.907988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.907878 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496"} err="failed to get container status \"5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496\": rpc error: code = NotFound desc = could not find container \"5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496\": container with ID starting with 5a1a5981bcccbc6bd90a1e5d616d60242d6cee03f5db3451de37583fc112f496 not found: ID does not exist" Oct 29 22:52:39.907988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.907895 5367 scope.go:122] "RemoveContainer" containerID="3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e" Oct 29 22:52:39.919303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.919284 5367 scope.go:122] "RemoveContainer" containerID="3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e" Oct 29 22:52:39.919681 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:39.919657 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e\": container with ID starting with 3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e not found: ID does not exist" containerID="3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e" Oct 29 22:52:39.919779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.919691 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e"} err="failed to get container status \"3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e\": rpc error: code = NotFound desc = could not find container \"3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e\": container with ID starting with 3749f4237461d1e42913e1b541f001b25753913576a83a1b235dd13612db513e not found: ID does not exist" Oct 29 22:52:39.951608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.951578 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl"] Oct 29 22:52:39.970622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.970588 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-379a6d-5d6bb4d488-59zfl"] Oct 29 22:52:39.993889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.993853 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw"] Oct 29 22:52:39.998664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:39.998640 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-379a6d-7487dfc7d7-xqnsw"] Oct 29 22:52:40.934916 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:40.934883 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t"] Oct 29 22:52:40.962681 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:40.962647 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz"] Oct 29 22:52:41.249845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:41.249770 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" path="/var/lib/kubelet/pods/7afc10ac-44a2-4b54-9f8d-f8111c94cc3a/volumes" Oct 29 22:52:41.250270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:41.250256 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" path="/var/lib/kubelet/pods/a4affabc-bf17-4f8e-a803-5a73d5485724/volumes" Oct 29 22:52:42.447848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:42.447814 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h"] Oct 29 22:52:42.482444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:42.482406 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd"] Oct 29 22:52:45.935317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:45.935277 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerName="payload-processing" containerID="cri-o://ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0" gracePeriod=25 Oct 29 22:52:45.963942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:45.963900 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerName="payload-pre-processing" containerID="cri-o://299b16042c6e366a21bb358204a645fb4eabeb7a170bd6e107ddfede1c6d743a" gracePeriod=25 Oct 29 22:52:47.448689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:47.448622 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerName="payload-processing" containerID="cri-o://d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9" gracePeriod=25 Oct 29 22:52:47.496176 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:47.496115 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerName="payload-pre-processing" containerID="cri-o://4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146" gracePeriod=25 Oct 29 22:52:49.667821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:49.667784 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2"] Oct 29 22:52:49.674455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:49.674411 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6bc7b2-nbzh2"] Oct 29 22:52:50.366404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:50.366370 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs"] Oct 29 22:52:50.378617 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:50.378585 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-172c10-5dvcs"] Oct 29 22:52:51.225147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:51.225116 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e79fa3c-9808-4d39-84a9-609a57c44e7d" path="/var/lib/kubelet/pods/2e79fa3c-9808-4d39-84a9-609a57c44e7d/volumes" Oct 29 22:52:51.225805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:51.225785 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" path="/var/lib/kubelet/pods/77fd9397-0ec3-4fb8-9c5c-55c31fe967a1/volumes" Oct 29 22:52:51.227100 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:51.227071 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:51.227200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:51.227109 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:52.331832 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:52:52.331797 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:52:52.499594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:52.499545 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:52.499801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:52.499557 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:52:57.033790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.033758 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7"] Oct 29 22:52:57.046384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.046357 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb7eca1-dqhc7"] Oct 29 22:52:57.152602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.152552 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld"] Oct 29 22:52:57.164966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.163670 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk"] Oct 29 22:52:57.164966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.164012 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" containerID="cri-o://028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa" gracePeriod=30 Oct 29 22:52:57.179658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.179591 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg"] Oct 29 22:52:57.182796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.179850 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" containerID="cri-o://410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a" gracePeriod=30 Oct 29 22:52:57.198293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.198240 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" containerID="cri-o://9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69" gracePeriod=30 Oct 29 22:52:57.224929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.224903 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2ffabb-8728-449b-8661-d3d18a0f50b5" path="/var/lib/kubelet/pods/ec2ffabb-8728-449b-8661-d3d18a0f50b5/volumes" Oct 29 22:52:57.677801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.677768 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q"] Oct 29 22:52:57.678745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678714 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerName="payload-processing" Oct 29 22:52:57.678745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678737 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerName="payload-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678754 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerName="payload-pre-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678762 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerName="payload-pre-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678785 5367 cpu_manager.go:414] "containerMap: removing container" podUID="77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" containerName="revoke-keys" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678792 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" containerName="revoke-keys" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678807 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerName="payload-pre-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678814 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerName="payload-pre-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678829 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerName="payload-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678872 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerName="payload-processing" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678889 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2f15f795-ffcc-4b85-8bbe-79491102b6cb" containerName="revoke-keys" Oct 29 22:52:57.678949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.678896 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f15f795-ffcc-4b85-8bbe-79491102b6cb" containerName="revoke-keys" Oct 29 22:52:57.679433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.679150 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4affabc-bf17-4f8e-a803-5a73d5485724" containerName="payload-processing" Oct 29 22:52:57.679433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.679164 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6b42d6-f6ad-4962-8a4c-2b49f4e101b9" containerName="payload-pre-processing" Oct 29 22:52:57.679433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.679177 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fd9397-0ec3-4fb8-9c5c-55c31fe967a1" containerName="revoke-keys" Oct 29 22:52:57.679433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.679189 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afc10ac-44a2-4b54-9f8d-f8111c94cc3a" containerName="payload-pre-processing" Oct 29 22:52:57.679433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.679200 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae5e8d4-de0b-4ecb-ae78-7495e9586400" containerName="payload-processing" Oct 29 22:52:57.679433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.679213 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f15f795-ffcc-4b85-8bbe-79491102b6cb" containerName="revoke-keys" Oct 29 22:52:57.684966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.684943 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.716666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.716635 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q"] Oct 29 22:52:57.752838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.752804 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt56z\" (UniqueName: \"kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.753044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.752866 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.753300 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.753271 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.761029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.761011 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:52:57.761167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.761142 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:52:57.854973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.854944 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt56z\" (UniqueName: \"kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.855142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.854994 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.855232 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.855216 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.855824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.855804 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.857470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.857447 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:57.875712 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:57.875674 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt56z\" (UniqueName: \"kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z\") pod \"maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:58.028409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.028317 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:52:58.377739 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.377710 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q"] Oct 29 22:52:58.422487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.422439 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x"] Oct 29 22:52:58.422809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.422715 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" containerID="cri-o://66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727" gracePeriod=30 Oct 29 22:52:58.853415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.853386 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87"] Oct 29 22:52:58.858347 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.858329 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:58.883110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.882193 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87"] Oct 29 22:52:58.975291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.975256 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:58.975464 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.975444 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj572\" (UniqueName: \"kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:58.975509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.975490 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:58.983899 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.983874 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:58.984011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.983924 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:58.992471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.992447 5367 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:58.992603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:58.992501 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:59.077585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.077550 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj572\" (UniqueName: \"kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.077762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.077599 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.077840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.077820 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.078338 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.078318 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.080121 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.080097 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.101576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.101551 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj572\" (UniqueName: \"kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572\") pod \"maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.159480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.159395 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:59.159640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.159476 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:59.189644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.189617 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:52:59.192924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.192901 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" event={"ID":"8fbea13e-a731-475d-b871-41bfb76bae2b","Type":"ContainerStarted","Data":"a497abc73e48192671aaadfd53b7306e1ff1879424ed3086ad9d3509abcb1a9d"} Oct 29 22:52:59.192991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.192955 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" event={"ID":"8fbea13e-a731-475d-b871-41bfb76bae2b","Type":"ContainerStarted","Data":"d50e7db5dfe5188c8ee32c112deb5bf5c028ad8011ce9322e8af4ae44639e187"} Oct 29 22:52:59.233387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.233354 5367 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:59.233550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.233411 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:59.240356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.240311 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" podStartSLOduration=2.24029879 podStartE2EDuration="2.24029879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:59.236691865 +0000 UTC m=+2246.130766788" watchObservedRunningTime="2026-10-29 22:52:59.24029879 +0000 UTC m=+2246.134373709" Oct 29 22:52:59.508404 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:52:59.508380 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe30707_d7af_4054_8d96_444f13daded1.slice/crio-0cab1e373b8acdba796a8bc597d48c263073eedf13364ba962272bc766e437a8 WatchSource:0}: Error finding container 0cab1e373b8acdba796a8bc597d48c263073eedf13364ba962272bc766e437a8: Status 404 returned error can't find the container with id 0cab1e373b8acdba796a8bc597d48c263073eedf13364ba962272bc766e437a8 Oct 29 22:52:59.509432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:52:59.509406 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87"] Oct 29 22:53:00.203577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:00.203490 5367 generic.go:356] "Generic (PLEG): container finished" podID="8fbea13e-a731-475d-b871-41bfb76bae2b" containerID="a497abc73e48192671aaadfd53b7306e1ff1879424ed3086ad9d3509abcb1a9d" exitCode=0 Oct 29 22:53:00.203796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:00.203581 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" event={"ID":"8fbea13e-a731-475d-b871-41bfb76bae2b","Type":"ContainerDied","Data":"a497abc73e48192671aaadfd53b7306e1ff1879424ed3086ad9d3509abcb1a9d"} Oct 29 22:53:00.205427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:00.205402 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" event={"ID":"cbe30707-d7af-4054-8d96-444f13daded1","Type":"ContainerStarted","Data":"d1ea17afdb066064f7b03222a4893d155e8d23cceaaa01ce4f18a8d1b77242ce"} Oct 29 22:53:00.205558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:00.205444 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" event={"ID":"cbe30707-d7af-4054-8d96-444f13daded1","Type":"ContainerStarted","Data":"0cab1e373b8acdba796a8bc597d48c263073eedf13364ba962272bc766e437a8"} Oct 29 22:53:00.318226 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:00.318177 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" podStartSLOduration=2.318163363 podStartE2EDuration="2.318163363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:00.313955374 +0000 UTC m=+2247.208030294" watchObservedRunningTime="2026-10-29 22:53:00.318163363 +0000 UTC m=+2247.212238282" Oct 29 22:53:01.216944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.216908 5367 generic.go:356] "Generic (PLEG): container finished" podID="cbe30707-d7af-4054-8d96-444f13daded1" containerID="d1ea17afdb066064f7b03222a4893d155e8d23cceaaa01ce4f18a8d1b77242ce" exitCode=0 Oct 29 22:53:01.226516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.226460 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" event={"ID":"cbe30707-d7af-4054-8d96-444f13daded1","Type":"ContainerDied","Data":"d1ea17afdb066064f7b03222a4893d155e8d23cceaaa01ce4f18a8d1b77242ce"} Oct 29 22:53:01.228052 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.228015 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:01.228186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.228137 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:01.402735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.402675 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:53:01.518188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.518111 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z\" (UniqueName: \"kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z\") pod \"8fbea13e-a731-475d-b871-41bfb76bae2b\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " Oct 29 22:53:01.518342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.518325 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca\") pod \"8fbea13e-a731-475d-b871-41bfb76bae2b\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " Oct 29 22:53:01.518389 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.518366 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca\") pod \"8fbea13e-a731-475d-b871-41bfb76bae2b\" (UID: \"8fbea13e-a731-475d-b871-41bfb76bae2b\") " Oct 29 22:53:01.518728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.518684 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca" pod "8fbea13e-a731-475d-b871-41bfb76bae2b" (UID: "8fbea13e-a731-475d-b871-41bfb76bae2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:01.519004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.518985 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:01.520265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.520244 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z" pod "8fbea13e-a731-475d-b871-41bfb76bae2b" (UID: "8fbea13e-a731-475d-b871-41bfb76bae2b"). InnerVolumeSpecName "kube-api-access-mt56z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:01.520519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.520500 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca" pod "8fbea13e-a731-475d-b871-41bfb76bae2b" (UID: "8fbea13e-a731-475d-b871-41bfb76bae2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:01.621104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.621065 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbea13e-a731-475d-b871-41bfb76bae2b-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:01.621104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:01.621101 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt56z\" (UniqueName: \"kubernetes.io/projected/8fbea13e-a731-475d-b871-41bfb76bae2b-kube-api-access-mt56z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.232318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.232296 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" Oct 29 22:53:02.232642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.232319 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q" event={"ID":"8fbea13e-a731-475d-b871-41bfb76bae2b","Type":"ContainerDied","Data":"d50e7db5dfe5188c8ee32c112deb5bf5c028ad8011ce9322e8af4ae44639e187"} Oct 29 22:53:02.232642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.232370 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50e7db5dfe5188c8ee32c112deb5bf5c028ad8011ce9322e8af4ae44639e187" Oct 29 22:53:02.428191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.428086 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:53:02.500179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.500140 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:02.502768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.501604 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:02.538414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.538383 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca\") pod \"cbe30707-d7af-4054-8d96-444f13daded1\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " Oct 29 22:53:02.538672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.538630 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572\" (UniqueName: \"kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572\") pod \"cbe30707-d7af-4054-8d96-444f13daded1\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " Oct 29 22:53:02.538785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.538677 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca\") pod \"cbe30707-d7af-4054-8d96-444f13daded1\" (UID: \"cbe30707-d7af-4054-8d96-444f13daded1\") " Oct 29 22:53:02.539960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.539910 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca" pod "cbe30707-d7af-4054-8d96-444f13daded1" (UID: "cbe30707-d7af-4054-8d96-444f13daded1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.569723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.566934 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572" pod "cbe30707-d7af-4054-8d96-444f13daded1" (UID: "cbe30707-d7af-4054-8d96-444f13daded1"). InnerVolumeSpecName "kube-api-access-xj572". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.570053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.570010 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca" pod "cbe30707-d7af-4054-8d96-444f13daded1" (UID: "cbe30707-d7af-4054-8d96-444f13daded1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:02.647768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.647679 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.647768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.647742 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj572\" (UniqueName: \"kubernetes.io/projected/cbe30707-d7af-4054-8d96-444f13daded1-kube-api-access-xj572\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.647988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.647782 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe30707-d7af-4054-8d96-444f13daded1-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.665721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.661342 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:53:02.672889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.672855 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:53:02.682380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.682353 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748290 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748350 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748412 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748567 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748607 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748677 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748758 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748825 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748856 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748885 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748943 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.748984 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749016 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749145 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749197 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749231 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749263 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749326 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749364 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749399 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749515 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749557 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749600 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy\") pod \"e858318b-4888-4949-8200-ccc0e1719ec6\" (UID: \"e858318b-4888-4949-8200-ccc0e1719ec6\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749624 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749658 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token\") pod \"ec53c939-317f-4bab-a39e-0c4e722e5050\" (UID: \"ec53c939-317f-4bab-a39e-0c4e722e5050\") " Oct 29 22:53:02.750752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.749728 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo\") pod \"530ad7df-e19e-4555-aab1-cc28df621b62\" (UID: \"530ad7df-e19e-4555-aab1-cc28df621b62\") " Oct 29 22:53:02.752382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.751810 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.752382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.752042 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.752382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.752346 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.753210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.752646 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.753210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.752941 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.753842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.753656 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.754330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.754123 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.754752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.754545 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.755030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.755002 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.755348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.755311 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.755575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.755554 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.755804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.755771 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.756986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.756384 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.756986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.756416 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.756986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.756222 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.756986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.756617 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.756986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.756911 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.757830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.757719 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.758610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.758583 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.763081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.760787 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.769436 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "kube-api-access-f4zkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.773648 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh" pod "530ad7df-e19e-4555-aab1-cc28df621b62" (UID: "530ad7df-e19e-4555-aab1-cc28df621b62"). InnerVolumeSpecName "kube-api-access-x89bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.773985 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.774104 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy" pod "ec53c939-317f-4bab-a39e-0c4e722e5050" (UID: "ec53c939-317f-4bab-a39e-0c4e722e5050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.774197 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "kube-api-access-g4zpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.777901 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.787621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.778043 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token" pod "e858318b-4888-4949-8200-ccc0e1719ec6" (UID: "e858318b-4888-4949-8200-ccc0e1719ec6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.853069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853033 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853063 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ad7df-e19e-4555-aab1-cc28df621b62-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853076 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853091 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853101 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89bh\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-kube-api-access-x89bh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853111 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ad7df-e19e-4555-aab1-cc28df621b62-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853119 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4zpv\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-kube-api-access-g4zpv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853127 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853138 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53c939-317f-4bab-a39e-0c4e722e5050-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853148 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853159 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853170 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ad7df-e19e-4555-aab1-cc28df621b62-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853181 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zkc\" (UniqueName: \"kubernetes.io/projected/ec53c939-317f-4bab-a39e-0c4e722e5050-kube-api-access-f4zkc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853193 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853207 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e858318b-4888-4949-8200-ccc0e1719ec6-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853218 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853231 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853244 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853257 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853270 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e858318b-4888-4949-8200-ccc0e1719ec6-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853283 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853295 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ad7df-e19e-4555-aab1-cc28df621b62-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853310 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853322 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853335 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e858318b-4888-4949-8200-ccc0e1719ec6-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853346 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e858318b-4888-4949-8200-ccc0e1719ec6-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.853432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:02.853357 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53c939-317f-4bab-a39e-0c4e722e5050-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.244738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.244678 5367 generic.go:356] "Generic (PLEG): container finished" podID="530ad7df-e19e-4555-aab1-cc28df621b62" containerID="410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a" exitCode=0 Oct 29 22:53:03.244738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.244726 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" event={"ID":"530ad7df-e19e-4555-aab1-cc28df621b62","Type":"ContainerDied","Data":"410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a"} Oct 29 22:53:03.245174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.244758 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" Oct 29 22:53:03.245174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.244773 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg" event={"ID":"530ad7df-e19e-4555-aab1-cc28df621b62","Type":"ContainerDied","Data":"6994ddeb3d754c9f95304512ed69ad76fe353f77b92eea6222031c1361cac791"} Oct 29 22:53:03.245174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.244803 5367 scope.go:122] "RemoveContainer" containerID="410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a" Oct 29 22:53:03.246400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.246381 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" Oct 29 22:53:03.246567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.246430 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87" event={"ID":"cbe30707-d7af-4054-8d96-444f13daded1","Type":"ContainerDied","Data":"0cab1e373b8acdba796a8bc597d48c263073eedf13364ba962272bc766e437a8"} Oct 29 22:53:03.246567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.246494 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cab1e373b8acdba796a8bc597d48c263073eedf13364ba962272bc766e437a8" Oct 29 22:53:03.248254 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.248231 5367 generic.go:356] "Generic (PLEG): container finished" podID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerID="028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa" exitCode=0 Oct 29 22:53:03.248342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.248260 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" Oct 29 22:53:03.248342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.248308 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" event={"ID":"ec53c939-317f-4bab-a39e-0c4e722e5050","Type":"ContainerDied","Data":"028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa"} Oct 29 22:53:03.248431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.248354 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk" event={"ID":"ec53c939-317f-4bab-a39e-0c4e722e5050","Type":"ContainerDied","Data":"f2e5874ff8bfcd7473255d754e0d9ac6eabe57e8064bfe4bc43c122f09115ab7"} Oct 29 22:53:03.250433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.250402 5367 generic.go:356] "Generic (PLEG): container finished" podID="e858318b-4888-4949-8200-ccc0e1719ec6" containerID="9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69" exitCode=0 Oct 29 22:53:03.250556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.250444 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" event={"ID":"e858318b-4888-4949-8200-ccc0e1719ec6","Type":"ContainerDied","Data":"9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69"} Oct 29 22:53:03.250556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.250452 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" Oct 29 22:53:03.250556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.250508 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld" event={"ID":"e858318b-4888-4949-8200-ccc0e1719ec6","Type":"ContainerDied","Data":"d97b9dd6757f9ea783a4f9e6aad75ce7bfe0cd11d5fa564e08a7236136f86318"} Oct 29 22:53:03.258972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.258953 5367 scope.go:122] "RemoveContainer" containerID="410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a" Oct 29 22:53:03.259227 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:03.259209 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a\": container with ID starting with 410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a not found: ID does not exist" containerID="410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a" Oct 29 22:53:03.259267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.259235 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a"} err="failed to get container status \"410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a\": rpc error: code = NotFound desc = could not find container \"410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a\": container with ID starting with 410acbdc254421ad32dc606ad96061494ffd100a3defc91c73c6cc7fea14561a not found: ID does not exist" Oct 29 22:53:03.259267 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.259252 5367 scope.go:122] "RemoveContainer" containerID="028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa" Oct 29 22:53:03.270439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.270414 5367 scope.go:122] "RemoveContainer" containerID="028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa" Oct 29 22:53:03.270756 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:03.270731 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa\": container with ID starting with 028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa not found: ID does not exist" containerID="028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa" Oct 29 22:53:03.270815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.270761 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa"} err="failed to get container status \"028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa\": rpc error: code = NotFound desc = could not find container \"028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa\": container with ID starting with 028941d648773b20d8f2af8acd387d172775fa574762431be1cf3c26dd2bb8fa not found: ID does not exist" Oct 29 22:53:03.270815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.270776 5367 scope.go:122] "RemoveContainer" containerID="9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69" Oct 29 22:53:03.281430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.281414 5367 scope.go:122] "RemoveContainer" containerID="cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938" Oct 29 22:53:03.291786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.291770 5367 scope.go:122] "RemoveContainer" containerID="9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69" Oct 29 22:53:03.292031 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:03.292009 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69\": container with ID starting with 9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69 not found: ID does not exist" containerID="9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69" Oct 29 22:53:03.292106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.292042 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69"} err="failed to get container status \"9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69\": rpc error: code = NotFound desc = could not find container \"9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69\": container with ID starting with 9e909be0e7dd23fb625daa6d5c7eaa2c9277fffdb577716351454145a5811c69 not found: ID does not exist" Oct 29 22:53:03.292106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.292066 5367 scope.go:122] "RemoveContainer" containerID="cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938" Oct 29 22:53:03.292299 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:03.292282 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938\": container with ID starting with cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938 not found: ID does not exist" containerID="cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938" Oct 29 22:53:03.292354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.292306 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938"} err="failed to get container status \"cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938\": rpc error: code = NotFound desc = could not find container \"cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938\": container with ID starting with cb507ac9f76885db56bcb7e38f33e42359bef02e2eb3ab32762fc9a4e991e938 not found: ID does not exist" Oct 29 22:53:03.308488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.308462 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg"] Oct 29 22:53:03.320766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.320741 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-sdnhg"] Oct 29 22:53:03.354179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.354154 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk"] Oct 29 22:53:03.358986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.358966 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-765tk"] Oct 29 22:53:03.399261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.399230 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld"] Oct 29 22:53:03.404087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.404064 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-txjld"] Oct 29 22:53:03.704074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.704049 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:53:03.767517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.767281 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.767772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.767751 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.768805 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.768666 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:03.769119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.769017 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.769279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.769254 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.769615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.769543 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.769862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.769827 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.771067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.770002 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.771067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.770042 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.771067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.770142 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.771067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.770192 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data\") pod \"2892e72b-9eb9-4ffa-abfa-a45306280418\" (UID: \"2892e72b-9eb9-4ffa-abfa-a45306280418\") " Oct 29 22:53:03.773234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.771208 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.773234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.771446 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.773234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.771498 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.773234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.771862 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:03.774063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.773967 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.774063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.773993 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.774063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.774006 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.774063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.774020 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2892e72b-9eb9-4ffa-abfa-a45306280418-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.774063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.774032 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.774063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.774043 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.774406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.774135 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:03.774406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.774178 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "kube-api-access-rshdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:03.790051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.790030 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy" pod "2892e72b-9eb9-4ffa-abfa-a45306280418" (UID: "2892e72b-9eb9-4ffa-abfa-a45306280418"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.875410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.875378 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.875410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.875408 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2892e72b-9eb9-4ffa-abfa-a45306280418-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.875611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:03.875420 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshdg\" (UniqueName: \"kubernetes.io/projected/2892e72b-9eb9-4ffa-abfa-a45306280418-kube-api-access-rshdg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:04.264358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.264274 5367 generic.go:356] "Generic (PLEG): container finished" podID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerID="66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727" exitCode=0 Oct 29 22:53:04.264358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.264321 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" event={"ID":"2892e72b-9eb9-4ffa-abfa-a45306280418","Type":"ContainerDied","Data":"66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727"} Oct 29 22:53:04.264822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.264361 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" event={"ID":"2892e72b-9eb9-4ffa-abfa-a45306280418","Type":"ContainerDied","Data":"0e1f8afaf40dc311821c6cf2465a45819c4b3498bc3d3eaa6a9bc8b9af0d18dd"} Oct 29 22:53:04.264822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.264383 5367 scope.go:122] "RemoveContainer" containerID="66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727" Oct 29 22:53:04.264822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.264323 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x" Oct 29 22:53:04.277011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.276995 5367 scope.go:122] "RemoveContainer" containerID="7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560" Oct 29 22:53:04.287316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.287298 5367 scope.go:122] "RemoveContainer" containerID="66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727" Oct 29 22:53:04.287562 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:04.287536 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727\": container with ID starting with 66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727 not found: ID does not exist" containerID="66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727" Oct 29 22:53:04.287605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.287568 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727"} err="failed to get container status \"66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727\": rpc error: code = NotFound desc = could not find container \"66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727\": container with ID starting with 66e15bee168f1b9beacd8aedf154c4d2adf9dd63b56f52abb425db61e6942727 not found: ID does not exist" Oct 29 22:53:04.287605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.287584 5367 scope.go:122] "RemoveContainer" containerID="7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560" Oct 29 22:53:04.287847 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:04.287826 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560\": container with ID starting with 7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560 not found: ID does not exist" containerID="7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560" Oct 29 22:53:04.287907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.287855 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560"} err="failed to get container status \"7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560\": rpc error: code = NotFound desc = could not find container \"7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560\": container with ID starting with 7d39de65352191bf27409225d47d69b4e6177537a65f85d86cd7fee4877d9560 not found: ID does not exist" Oct 29 22:53:04.357517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.357477 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x"] Oct 29 22:53:04.378186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:04.375140 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-zm58x"] Oct 29 22:53:05.240918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:05.240848 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" path="/var/lib/kubelet/pods/2892e72b-9eb9-4ffa-abfa-a45306280418/volumes" Oct 29 22:53:05.242022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:05.241991 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" path="/var/lib/kubelet/pods/530ad7df-e19e-4555-aab1-cc28df621b62/volumes" Oct 29 22:53:05.242753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:05.242693 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" path="/var/lib/kubelet/pods/e858318b-4888-4949-8200-ccc0e1719ec6/volumes" Oct 29 22:53:05.243391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:05.243375 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" path="/var/lib/kubelet/pods/ec53c939-317f-4bab-a39e-0c4e722e5050/volumes" Oct 29 22:53:06.271898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.271864 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:53:06.290163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.290075 5367 generic.go:356] "Generic (PLEG): container finished" podID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerID="299b16042c6e366a21bb358204a645fb4eabeb7a170bd6e107ddfede1c6d743a" exitCode=0 Oct 29 22:53:06.290163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.290105 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" event={"ID":"dc054d32-b8a2-4613-aeaf-37c1c98b0862","Type":"ContainerDied","Data":"299b16042c6e366a21bb358204a645fb4eabeb7a170bd6e107ddfede1c6d743a"} Oct 29 22:53:06.292000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.291639 5367 generic.go:356] "Generic (PLEG): container finished" podID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerID="ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0" exitCode=0 Oct 29 22:53:06.292000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.291720 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" Oct 29 22:53:06.292000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.291730 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" event={"ID":"cc928179-f6f2-4cc6-a4ad-c5653601e838","Type":"ContainerDied","Data":"ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0"} Oct 29 22:53:06.292000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.291781 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t" event={"ID":"cc928179-f6f2-4cc6-a4ad-c5653601e838","Type":"ContainerDied","Data":"289fbde98de3892464ab3b11bf3084ddd478d5ce2838a98bb694810c1a1dd567"} Oct 29 22:53:06.292000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.291809 5367 scope.go:122] "RemoveContainer" containerID="ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0" Oct 29 22:53:06.305078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.305064 5367 scope.go:122] "RemoveContainer" containerID="ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0" Oct 29 22:53:06.305350 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:06.305330 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0\": container with ID starting with ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0 not found: ID does not exist" containerID="ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0" Oct 29 22:53:06.305428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.305359 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0"} err="failed to get container status \"ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0\": rpc error: code = NotFound desc = could not find container \"ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0\": container with ID starting with ab4cdeda020a1b611777af8fe573cbc5c028c2ba80803375a96fe69b1a173eb0 not found: ID does not exist" Oct 29 22:53:06.307002 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.306985 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:53:06.416241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.416211 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78\" (UniqueName: \"kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78\") pod \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " Oct 29 22:53:06.416516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.416250 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume\") pod \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\" (UID: \"dc054d32-b8a2-4613-aeaf-37c1c98b0862\") " Oct 29 22:53:06.416516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.416311 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv\" (UniqueName: \"kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv\") pod \"cc928179-f6f2-4cc6-a4ad-c5653601e838\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " Oct 29 22:53:06.416516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.416481 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume\") pod \"cc928179-f6f2-4cc6-a4ad-c5653601e838\" (UID: \"cc928179-f6f2-4cc6-a4ad-c5653601e838\") " Oct 29 22:53:06.416864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.416836 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume" pod "dc054d32-b8a2-4613-aeaf-37c1c98b0862" (UID: "dc054d32-b8a2-4613-aeaf-37c1c98b0862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:06.416959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.416892 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume" pod "cc928179-f6f2-4cc6-a4ad-c5653601e838" (UID: "cc928179-f6f2-4cc6-a4ad-c5653601e838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:06.417522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.417498 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc054d32-b8a2-4613-aeaf-37c1c98b0862-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:06.417522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.417521 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc928179-f6f2-4cc6-a4ad-c5653601e838-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:06.419006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.418984 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78" pod "dc054d32-b8a2-4613-aeaf-37c1c98b0862" (UID: "dc054d32-b8a2-4613-aeaf-37c1c98b0862"). InnerVolumeSpecName "kube-api-access-cfj78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:06.419265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.419228 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv" pod "cc928179-f6f2-4cc6-a4ad-c5653601e838" (UID: "cc928179-f6f2-4cc6-a4ad-c5653601e838"). InnerVolumeSpecName "kube-api-access-j5blv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:06.519137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.519073 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5blv\" (UniqueName: \"kubernetes.io/projected/cc928179-f6f2-4cc6-a4ad-c5653601e838-kube-api-access-j5blv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:06.519137 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.519099 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfj78\" (UniqueName: \"kubernetes.io/projected/dc054d32-b8a2-4613-aeaf-37c1c98b0862-kube-api-access-cfj78\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:06.644264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.644233 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t"] Oct 29 22:53:06.656936 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:06.656908 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad69b4-7dc6574fc8-8tp7t"] Oct 29 22:53:07.228194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.228158 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" path="/var/lib/kubelet/pods/cc928179-f6f2-4cc6-a4ad-c5653601e838/volumes" Oct 29 22:53:07.303920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.303890 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" Oct 29 22:53:07.303920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.303905 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz" event={"ID":"dc054d32-b8a2-4613-aeaf-37c1c98b0862","Type":"ContainerDied","Data":"e711c0cd9aeeac763e0d5113b18bc3ee31c9fc40eb24f26de274b0381fbf9297"} Oct 29 22:53:07.304426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.303975 5367 scope.go:122] "RemoveContainer" containerID="299b16042c6e366a21bb358204a645fb4eabeb7a170bd6e107ddfede1c6d743a" Oct 29 22:53:07.361800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.361764 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz"] Oct 29 22:53:07.374998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.374961 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad69b4-6c5649b4b7-bkmwz"] Oct 29 22:53:07.832733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.831872 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:53:07.846434 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.846400 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:53:07.947634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.947542 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv\" (UniqueName: \"kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv\") pod \"677161a8-a4f8-4c50-93aa-3a73b8818297\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " Oct 29 22:53:07.947634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.947627 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw\" (UniqueName: \"kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw\") pod \"1646498f-db1a-4e03-8701-7c02e75d1f33\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " Oct 29 22:53:07.947943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.947725 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume\") pod \"677161a8-a4f8-4c50-93aa-3a73b8818297\" (UID: \"677161a8-a4f8-4c50-93aa-3a73b8818297\") " Oct 29 22:53:07.947943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.947810 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume\") pod \"1646498f-db1a-4e03-8701-7c02e75d1f33\" (UID: \"1646498f-db1a-4e03-8701-7c02e75d1f33\") " Oct 29 22:53:07.948333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.948309 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume" pod "677161a8-a4f8-4c50-93aa-3a73b8818297" (UID: "677161a8-a4f8-4c50-93aa-3a73b8818297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:07.948450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.948410 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume" pod "1646498f-db1a-4e03-8701-7c02e75d1f33" (UID: "1646498f-db1a-4e03-8701-7c02e75d1f33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:07.948867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.948666 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677161a8-a4f8-4c50-93aa-3a73b8818297-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:07.948867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.948688 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1646498f-db1a-4e03-8701-7c02e75d1f33-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:07.952397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.951978 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw" pod "1646498f-db1a-4e03-8701-7c02e75d1f33" (UID: "1646498f-db1a-4e03-8701-7c02e75d1f33"). InnerVolumeSpecName "kube-api-access-zm9nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:07.952634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:07.952612 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv" pod "677161a8-a4f8-4c50-93aa-3a73b8818297" (UID: "677161a8-a4f8-4c50-93aa-3a73b8818297"). InnerVolumeSpecName "kube-api-access-fhqnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:08.128746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.110228 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqnv\" (UniqueName: \"kubernetes.io/projected/677161a8-a4f8-4c50-93aa-3a73b8818297-kube-api-access-fhqnv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:08.128746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.110266 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9nw\" (UniqueName: \"kubernetes.io/projected/1646498f-db1a-4e03-8701-7c02e75d1f33-kube-api-access-zm9nw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:08.274586 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.274500 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm"] Oct 29 22:53:08.311745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.311687 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx"] Oct 29 22:53:08.317581 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.317552 5367 generic.go:356] "Generic (PLEG): container finished" podID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerID="4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146" exitCode=0 Oct 29 22:53:08.317834 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.317815 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" event={"ID":"1646498f-db1a-4e03-8701-7c02e75d1f33","Type":"ContainerDied","Data":"4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146"} Oct 29 22:53:08.317931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.317867 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" event={"ID":"1646498f-db1a-4e03-8701-7c02e75d1f33","Type":"ContainerDied","Data":"19308a26a7489ddbb0d84ee272cc460ab70bbb524209b450020bbe5a28c7d874"} Oct 29 22:53:08.317931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.317890 5367 scope.go:122] "RemoveContainer" containerID="4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146" Oct 29 22:53:08.318043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.317817 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd" Oct 29 22:53:08.322057 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.322027 5367 generic.go:356] "Generic (PLEG): container finished" podID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerID="d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9" exitCode=0 Oct 29 22:53:08.322202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.322079 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" Oct 29 22:53:08.322202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.322116 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" event={"ID":"677161a8-a4f8-4c50-93aa-3a73b8818297","Type":"ContainerDied","Data":"d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9"} Oct 29 22:53:08.322202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.322180 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h" event={"ID":"677161a8-a4f8-4c50-93aa-3a73b8818297","Type":"ContainerDied","Data":"97d53e7fda3c5a78330266083d65587fe613fde554a02351653d60f6edc7f5fb"} Oct 29 22:53:08.334056 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.334038 5367 scope.go:122] "RemoveContainer" containerID="4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146" Oct 29 22:53:08.334331 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:08.334313 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146\": container with ID starting with 4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146 not found: ID does not exist" containerID="4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146" Oct 29 22:53:08.334400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.334340 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146"} err="failed to get container status \"4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146\": rpc error: code = NotFound desc = could not find container \"4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146\": container with ID starting with 4ccb4079775568986c769f40682b3c5b128da5cdd9c5328123b630a760a7a146 not found: ID does not exist" Oct 29 22:53:08.334488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.334401 5367 scope.go:122] "RemoveContainer" containerID="d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9" Oct 29 22:53:08.346096 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.346077 5367 scope.go:122] "RemoveContainer" containerID="d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9" Oct 29 22:53:08.346360 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:08.346342 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9\": container with ID starting with d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9 not found: ID does not exist" containerID="d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9" Oct 29 22:53:08.346406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.346370 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9"} err="failed to get container status \"d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9\": rpc error: code = NotFound desc = could not find container \"d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9\": container with ID starting with d57b7cf114d8b717f5a789bdab84ad6b85767c84aecf75af4e94aa37462c7fb9 not found: ID does not exist" Oct 29 22:53:08.401835 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.401805 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd"] Oct 29 22:53:08.411883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.411856 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-172c10-6b9f9f49ff-2tsmd"] Oct 29 22:53:08.449188 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.449157 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h"] Oct 29 22:53:08.465768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:08.454962 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-172c10-57c45696d8-9697h"] Oct 29 22:53:09.320400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:09.320363 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" path="/var/lib/kubelet/pods/1646498f-db1a-4e03-8701-7c02e75d1f33/volumes" Oct 29 22:53:09.411631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:09.411593 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" path="/var/lib/kubelet/pods/677161a8-a4f8-4c50-93aa-3a73b8818297/volumes" Oct 29 22:53:09.439722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:09.431837 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" path="/var/lib/kubelet/pods/dc054d32-b8a2-4613-aeaf-37c1c98b0862/volumes" Oct 29 22:53:10.164605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:10.164133 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9"] Oct 29 22:53:10.180268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:10.180230 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad69b4-g9cv9"] Oct 29 22:53:10.389759 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:10.389723 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv"] Oct 29 22:53:10.427097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:10.427030 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq"] Oct 29 22:53:11.224919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:11.224885 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f15f795-ffcc-4b85-8bbe-79491102b6cb" path="/var/lib/kubelet/pods/2f15f795-ffcc-4b85-8bbe-79491102b6cb/volumes" Oct 29 22:53:12.120686 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.120651 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c"] Oct 29 22:53:12.121418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121401 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8fbea13e-a731-475d-b871-41bfb76bae2b" containerName="revoke-keys" Oct 29 22:53:12.121463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121420 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbea13e-a731-475d-b871-41bfb76bae2b" containerName="revoke-keys" Oct 29 22:53:12.121463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121433 5367 cpu_manager.go:414] "containerMap: removing container" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerName="payload-processing" Oct 29 22:53:12.121463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121439 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerName="payload-processing" Oct 29 22:53:12.121463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121454 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" Oct 29 22:53:12.121463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121459 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121469 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121474 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121484 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerName="payload-pre-processing" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121490 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerName="payload-pre-processing" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121496 5367 cpu_manager.go:414] "containerMap: removing container" podUID="cbe30707-d7af-4054-8d96-444f13daded1" containerName="revoke-keys" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121500 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe30707-d7af-4054-8d96-444f13daded1" containerName="revoke-keys" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121511 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121515 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121534 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121538 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121553 5367 cpu_manager.go:414] "containerMap: removing container" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerName="payload-processing" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121557 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerName="payload-processing" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121567 5367 cpu_manager.go:414] "containerMap: removing container" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerName="payload-pre-processing" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121571 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerName="payload-pre-processing" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121582 5367 cpu_manager.go:414] "containerMap: removing container" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" Oct 29 22:53:12.121628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121586 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121683 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121693 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121710 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="677161a8-a4f8-4c50-93aa-3a73b8818297" containerName="payload-processing" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121717 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc054d32-b8a2-4613-aeaf-37c1c98b0862" containerName="payload-pre-processing" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121723 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="530ad7df-e19e-4555-aab1-cc28df621b62" containerName="istio-proxy" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121730 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbea13e-a731-475d-b871-41bfb76bae2b" containerName="revoke-keys" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121735 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc928179-f6f2-4cc6-a4ad-c5653601e838" containerName="payload-processing" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121743 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec53c939-317f-4bab-a39e-0c4e722e5050" containerName="istio-proxy" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121749 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1646498f-db1a-4e03-8701-7c02e75d1f33" containerName="payload-pre-processing" Oct 29 22:53:12.122172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.121754 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe30707-d7af-4054-8d96-444f13daded1" containerName="revoke-keys" Oct 29 22:53:12.126752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.126736 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.146211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.146186 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c"] Oct 29 22:53:12.152770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.152745 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:12.152917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.152806 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:53:12.186386 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.186355 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.186587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.186571 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpd2\" (UniqueName: \"kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.186768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.186744 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.288538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.288499 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.288763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.288629 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.288763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.288736 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpd2\" (UniqueName: \"kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.289330 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.289304 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.290902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.290878 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.309429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.309408 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpd2\" (UniqueName: \"kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2\") pod \"maas-api-revoke-sub-6a03bfd5-sth5c\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.309563 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.309544 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx"] Oct 29 22:53:12.310754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.310733 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" Oct 29 22:53:12.310822 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.310757 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" Oct 29 22:53:12.311015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.311003 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e858318b-4888-4949-8200-ccc0e1719ec6" containerName="istio-proxy" Oct 29 22:53:12.311053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.311037 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2892e72b-9eb9-4ffa-abfa-a45306280418" containerName="istio-proxy" Oct 29 22:53:12.315924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.315905 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.336653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.336626 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx"] Oct 29 22:53:12.390621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.390551 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.390788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.390735 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.390788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.390768 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.441283 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.441253 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:12.492205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.492120 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.492483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.492332 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.492804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.492366 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.493429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.493272 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.494860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.494834 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.513245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.513225 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf\") pod \"maas-api-revoke-sub-bac6b1fd-mjcmx\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.666489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.666382 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:12.759928 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.759678 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c"] Oct 29 22:53:12.763351 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:53:12.763312 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f08ea58_e238_47f6_b054_7795c8916a2e.slice/crio-bf09a9c8a88c0b40d0c9babaa94db51e298e90719f1e16140ff04b23cbfb8f64 WatchSource:0}: Error finding container bf09a9c8a88c0b40d0c9babaa94db51e298e90719f1e16140ff04b23cbfb8f64: Status 404 returned error can't find the container with id bf09a9c8a88c0b40d0c9babaa94db51e298e90719f1e16140ff04b23cbfb8f64 Oct 29 22:53:12.974232 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:53:12.974201 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417c6534_7643_49d6_a44d_de910b7978ba.slice/crio-fa172af236c99f9bb091288f4d790dc9791d3f3a9b1773cea405a0164a98ea05 WatchSource:0}: Error finding container fa172af236c99f9bb091288f4d790dc9791d3f3a9b1773cea405a0164a98ea05: Status 404 returned error can't find the container with id fa172af236c99f9bb091288f4d790dc9791d3f3a9b1773cea405a0164a98ea05 Oct 29 22:53:12.975218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:12.975172 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx"] Oct 29 22:53:13.275830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.275803 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" podUID="859b88c4-18d2-4c33-936d-b5b20a2a32f0" containerName="payload-processing" containerID="cri-o://b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d" gracePeriod=25 Oct 29 22:53:13.312333 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.312290 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerName="payload-pre-processing" containerID="cri-o://7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04" gracePeriod=25 Oct 29 22:53:13.574861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.574834 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:53:13.601045 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.601016 5367 generic.go:356] "Generic (PLEG): container finished" podID="859b88c4-18d2-4c33-936d-b5b20a2a32f0" containerID="b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d" exitCode=0 Oct 29 22:53:13.601203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.601060 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" Oct 29 22:53:13.601203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.601101 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" event={"ID":"859b88c4-18d2-4c33-936d-b5b20a2a32f0","Type":"ContainerDied","Data":"b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d"} Oct 29 22:53:13.601203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.601159 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm" event={"ID":"859b88c4-18d2-4c33-936d-b5b20a2a32f0","Type":"ContainerDied","Data":"09c03f7e3549e77aaa9df25ac656efd0945ce1c764989e0e11d3c41aa6f41f2b"} Oct 29 22:53:13.601203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.601192 5367 scope.go:122] "RemoveContainer" containerID="b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d" Oct 29 22:53:13.602989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.602961 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" event={"ID":"9f08ea58-e238-47f6-b054-7795c8916a2e","Type":"ContainerStarted","Data":"21d1b38735e6a49b52be630f6280cd4ebe1b83d2989b8e0f89cc4355af661e33"} Oct 29 22:53:13.603097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.603014 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" event={"ID":"9f08ea58-e238-47f6-b054-7795c8916a2e","Type":"ContainerStarted","Data":"bf09a9c8a88c0b40d0c9babaa94db51e298e90719f1e16140ff04b23cbfb8f64"} Oct 29 22:53:13.604877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.604855 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" event={"ID":"417c6534-7643-49d6-a44d-de910b7978ba","Type":"ContainerStarted","Data":"be4fcd96e2574d75a031804befacbeaf49ec93b38e7d77733d403f1819c2fb33"} Oct 29 22:53:13.604983 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.604895 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" event={"ID":"417c6534-7643-49d6-a44d-de910b7978ba","Type":"ContainerStarted","Data":"fa172af236c99f9bb091288f4d790dc9791d3f3a9b1773cea405a0164a98ea05"} Oct 29 22:53:13.609736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.609714 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume\") pod \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " Oct 29 22:53:13.609856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.609839 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj\" (UniqueName: \"kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj\") pod \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\" (UID: \"859b88c4-18d2-4c33-936d-b5b20a2a32f0\") " Oct 29 22:53:13.609980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.609961 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume" pod "859b88c4-18d2-4c33-936d-b5b20a2a32f0" (UID: "859b88c4-18d2-4c33-936d-b5b20a2a32f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:13.610574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.610553 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859b88c4-18d2-4c33-936d-b5b20a2a32f0-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:13.611852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.611829 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj" pod "859b88c4-18d2-4c33-936d-b5b20a2a32f0" (UID: "859b88c4-18d2-4c33-936d-b5b20a2a32f0"). InnerVolumeSpecName "kube-api-access-xzxnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:13.613487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.613464 5367 scope.go:122] "RemoveContainer" containerID="b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d" Oct 29 22:53:13.613772 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:13.613746 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d\": container with ID starting with b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d not found: ID does not exist" containerID="b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d" Oct 29 22:53:13.613838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.613778 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d"} err="failed to get container status \"b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d\": rpc error: code = NotFound desc = could not find container \"b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d\": container with ID starting with b428ec1cce2524732f021e188f28df6c62a7f6a3a698726b0a3b0caaefbb566d not found: ID does not exist" Oct 29 22:53:13.714625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.714543 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzxnj\" (UniqueName: \"kubernetes.io/projected/859b88c4-18d2-4c33-936d-b5b20a2a32f0-kube-api-access-xzxnj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:13.719973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.719911 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" podStartSLOduration=1.71989323 podStartE2EDuration="1.71989323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:13.712601129 +0000 UTC m=+2260.606676049" watchObservedRunningTime="2026-10-29 22:53:13.71989323 +0000 UTC m=+2260.613968152" Oct 29 22:53:13.775373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.775329 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" podStartSLOduration=1.775313379 podStartE2EDuration="1.775313379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:13.773643809 +0000 UTC m=+2260.667718729" watchObservedRunningTime="2026-10-29 22:53:13.775313379 +0000 UTC m=+2260.669388322" Oct 29 22:53:13.991024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.990962 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm"] Oct 29 22:53:13.995829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:13.995805 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dd4c8-bbd854d48-b8nkm"] Oct 29 22:53:14.616997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:14.616918 5367 generic.go:356] "Generic (PLEG): container finished" podID="9f08ea58-e238-47f6-b054-7795c8916a2e" containerID="21d1b38735e6a49b52be630f6280cd4ebe1b83d2989b8e0f89cc4355af661e33" exitCode=0 Oct 29 22:53:14.616997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:14.616969 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" event={"ID":"9f08ea58-e238-47f6-b054-7795c8916a2e","Type":"ContainerDied","Data":"21d1b38735e6a49b52be630f6280cd4ebe1b83d2989b8e0f89cc4355af661e33"} Oct 29 22:53:14.618675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:14.618648 5367 generic.go:356] "Generic (PLEG): container finished" podID="417c6534-7643-49d6-a44d-de910b7978ba" containerID="be4fcd96e2574d75a031804befacbeaf49ec93b38e7d77733d403f1819c2fb33" exitCode=0 Oct 29 22:53:14.618800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:14.618726 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" event={"ID":"417c6534-7643-49d6-a44d-de910b7978ba","Type":"ContainerDied","Data":"be4fcd96e2574d75a031804befacbeaf49ec93b38e7d77733d403f1819c2fb33"} Oct 29 22:53:15.224533 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.224503 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859b88c4-18d2-4c33-936d-b5b20a2a32f0" path="/var/lib/kubelet/pods/859b88c4-18d2-4c33-936d-b5b20a2a32f0/volumes" Oct 29 22:53:15.390243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.390199 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" podUID="61c670f9-76f0-4d6a-a9a2-e2373fb09f83" containerName="payload-processing" containerID="cri-o://9409a4ebf0acf854a03009ea7ec72677a22af3efb3915aa467fb3cf0ba07918c" gracePeriod=25 Oct 29 22:53:15.427820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.427783 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" containerName="payload-pre-processing" containerID="cri-o://ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9" gracePeriod=25 Oct 29 22:53:15.638190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.638153 5367 generic.go:356] "Generic (PLEG): container finished" podID="61c670f9-76f0-4d6a-a9a2-e2373fb09f83" containerID="9409a4ebf0acf854a03009ea7ec72677a22af3efb3915aa467fb3cf0ba07918c" exitCode=0 Oct 29 22:53:15.638564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.638227 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" event={"ID":"61c670f9-76f0-4d6a-a9a2-e2373fb09f83","Type":"ContainerDied","Data":"9409a4ebf0acf854a03009ea7ec72677a22af3efb3915aa467fb3cf0ba07918c"} Oct 29 22:53:15.690744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.687335 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:53:15.753783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.753751 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume\") pod \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " Oct 29 22:53:15.753989 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.753974 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb\" (UniqueName: \"kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb\") pod \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\" (UID: \"61c670f9-76f0-4d6a-a9a2-e2373fb09f83\") " Oct 29 22:53:15.754207 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.754180 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume" pod "61c670f9-76f0-4d6a-a9a2-e2373fb09f83" (UID: "61c670f9-76f0-4d6a-a9a2-e2373fb09f83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:15.754724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.754685 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:15.756900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.756871 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb" pod "61c670f9-76f0-4d6a-a9a2-e2373fb09f83" (UID: "61c670f9-76f0-4d6a-a9a2-e2373fb09f83"). InnerVolumeSpecName "kube-api-access-75wlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:15.827225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.827192 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:15.844308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.844282 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:15.855972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.855951 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf\") pod \"417c6534-7643-49d6-a44d-de910b7978ba\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " Oct 29 22:53:15.856101 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.856038 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca\") pod \"417c6534-7643-49d6-a44d-de910b7978ba\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " Oct 29 22:53:15.856168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.856136 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca\") pod \"417c6534-7643-49d6-a44d-de910b7978ba\" (UID: \"417c6534-7643-49d6-a44d-de910b7978ba\") " Oct 29 22:53:15.856458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.856434 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca" pod "417c6534-7643-49d6-a44d-de910b7978ba" (UID: "417c6534-7643-49d6-a44d-de910b7978ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:15.856732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.856686 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:15.856806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.856740 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wlb\" (UniqueName: \"kubernetes.io/projected/61c670f9-76f0-4d6a-a9a2-e2373fb09f83-kube-api-access-75wlb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:15.858019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.857998 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca" pod "417c6534-7643-49d6-a44d-de910b7978ba" (UID: "417c6534-7643-49d6-a44d-de910b7978ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:15.858094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.858079 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf" pod "417c6534-7643-49d6-a44d-de910b7978ba" (UID: "417c6534-7643-49d6-a44d-de910b7978ba"). InnerVolumeSpecName "kube-api-access-hp5tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:15.958220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.958187 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2\" (UniqueName: \"kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2\") pod \"9f08ea58-e238-47f6-b054-7795c8916a2e\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " Oct 29 22:53:15.958427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.958373 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca\") pod \"9f08ea58-e238-47f6-b054-7795c8916a2e\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " Oct 29 22:53:15.958795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.958768 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca" pod "9f08ea58-e238-47f6-b054-7795c8916a2e" (UID: "9f08ea58-e238-47f6-b054-7795c8916a2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:15.959072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.959049 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca\") pod \"9f08ea58-e238-47f6-b054-7795c8916a2e\" (UID: \"9f08ea58-e238-47f6-b054-7795c8916a2e\") " Oct 29 22:53:15.959815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.959792 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:15.959949 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.959933 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/417c6534-7643-49d6-a44d-de910b7978ba-kube-api-access-hp5tf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:15.960061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.960048 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c6534-7643-49d6-a44d-de910b7978ba-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:15.960978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.960953 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2" pod "9f08ea58-e238-47f6-b054-7795c8916a2e" (UID: "9f08ea58-e238-47f6-b054-7795c8916a2e"). InnerVolumeSpecName "kube-api-access-wvpd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:15.962230 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:15.962188 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca" pod "9f08ea58-e238-47f6-b054-7795c8916a2e" (UID: "9f08ea58-e238-47f6-b054-7795c8916a2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:16.035889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.035805 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm"] Oct 29 22:53:16.041866 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.041839 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6636e3f-z2wdm"] Oct 29 22:53:16.062120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.062094 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f08ea58-e238-47f6-b054-7795c8916a2e-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:16.062120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.062120 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpd2\" (UniqueName: \"kubernetes.io/projected/9f08ea58-e238-47f6-b054-7795c8916a2e-kube-api-access-wvpd2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:16.648828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.648800 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" Oct 29 22:53:16.649271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.648841 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv" event={"ID":"61c670f9-76f0-4d6a-a9a2-e2373fb09f83","Type":"ContainerDied","Data":"16fa148241bbea1e375c608c9a7d1071583da35fcdd3f66e87ee89932c6c33cd"} Oct 29 22:53:16.649271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.648905 5367 scope.go:122] "RemoveContainer" containerID="9409a4ebf0acf854a03009ea7ec72677a22af3efb3915aa467fb3cf0ba07918c" Oct 29 22:53:16.650662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.650645 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" Oct 29 22:53:16.650829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.650686 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c" event={"ID":"9f08ea58-e238-47f6-b054-7795c8916a2e","Type":"ContainerDied","Data":"bf09a9c8a88c0b40d0c9babaa94db51e298e90719f1e16140ff04b23cbfb8f64"} Oct 29 22:53:16.650829 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.650773 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf09a9c8a88c0b40d0c9babaa94db51e298e90719f1e16140ff04b23cbfb8f64" Oct 29 22:53:16.652458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.652439 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" Oct 29 22:53:16.652732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.652463 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx" event={"ID":"417c6534-7643-49d6-a44d-de910b7978ba","Type":"ContainerDied","Data":"fa172af236c99f9bb091288f4d790dc9791d3f3a9b1773cea405a0164a98ea05"} Oct 29 22:53:16.652839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.652772 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa172af236c99f9bb091288f4d790dc9791d3f3a9b1773cea405a0164a98ea05" Oct 29 22:53:16.719476 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.719448 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv"] Oct 29 22:53:16.732747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:16.732720 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8ebc60-58cbf955c8-7swqv"] Oct 29 22:53:17.238723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:17.238313 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1786c820-4fda-4b71-9490-d207be700219" path="/var/lib/kubelet/pods/1786c820-4fda-4b71-9490-d207be700219/volumes" Oct 29 22:53:17.245818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:17.238989 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c670f9-76f0-4d6a-a9a2-e2373fb09f83" path="/var/lib/kubelet/pods/61c670f9-76f0-4d6a-a9a2-e2373fb09f83/volumes" Oct 29 22:53:17.414934 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:17.414900 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn"] Oct 29 22:53:17.415195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:17.415159 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" containerID="cri-o://afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75" gracePeriod=30 Oct 29 22:53:19.657353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:19.657316 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:22.337495 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:22.337451 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:53:22.549200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.549171 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm"] Oct 29 22:53:22.550001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.549984 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9f08ea58-e238-47f6-b054-7795c8916a2e" containerName="revoke-keys" Oct 29 22:53:22.550064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550003 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f08ea58-e238-47f6-b054-7795c8916a2e" containerName="revoke-keys" Oct 29 22:53:22.550064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550024 5367 cpu_manager.go:414] "containerMap: removing container" podUID="61c670f9-76f0-4d6a-a9a2-e2373fb09f83" containerName="payload-processing" Oct 29 22:53:22.550064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550030 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c670f9-76f0-4d6a-a9a2-e2373fb09f83" containerName="payload-processing" Oct 29 22:53:22.550064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550047 5367 cpu_manager.go:414] "containerMap: removing container" podUID="417c6534-7643-49d6-a44d-de910b7978ba" containerName="revoke-keys" Oct 29 22:53:22.550064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550053 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417c6534-7643-49d6-a44d-de910b7978ba" containerName="revoke-keys" Oct 29 22:53:22.550239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550067 5367 cpu_manager.go:414] "containerMap: removing container" podUID="859b88c4-18d2-4c33-936d-b5b20a2a32f0" containerName="payload-processing" Oct 29 22:53:22.550239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550071 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859b88c4-18d2-4c33-936d-b5b20a2a32f0" containerName="payload-processing" Oct 29 22:53:22.550239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550185 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="859b88c4-18d2-4c33-936d-b5b20a2a32f0" containerName="payload-processing" Oct 29 22:53:22.550239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550199 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f08ea58-e238-47f6-b054-7795c8916a2e" containerName="revoke-keys" Oct 29 22:53:22.550239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550207 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c670f9-76f0-4d6a-a9a2-e2373fb09f83" containerName="payload-processing" Oct 29 22:53:22.550239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.550217 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="417c6534-7643-49d6-a44d-de910b7978ba" containerName="revoke-keys" Oct 29 22:53:22.554128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.554088 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.577735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.577690 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm"] Oct 29 22:53:22.581879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.581853 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:53:22.582032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.581942 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:22.655168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.655134 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms"] Oct 29 22:53:22.661126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.661097 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.675142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.675113 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.675278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.675202 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.675356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.675339 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmf75\" (UniqueName: \"kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.685738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.684097 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms"] Oct 29 22:53:22.748227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.748202 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:53:22.760518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.760489 5367 generic.go:356] "Generic (PLEG): container finished" podID="275ee01a-c16a-42f8-b332-e2728b091f71" containerID="afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75" exitCode=0 Oct 29 22:53:22.760642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.760540 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" Oct 29 22:53:22.760793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.760768 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" event={"ID":"275ee01a-c16a-42f8-b332-e2728b091f71","Type":"ContainerDied","Data":"afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75"} Oct 29 22:53:22.760864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.760836 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn" event={"ID":"275ee01a-c16a-42f8-b332-e2728b091f71","Type":"ContainerDied","Data":"0b7dd74d7eca88f49f1f407a41f6abb528ba12fe48427169f0c63dfa50f8f20e"} Oct 29 22:53:22.760920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.760874 5367 scope.go:122] "RemoveContainer" containerID="afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75" Oct 29 22:53:22.777621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.777598 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmf75\" (UniqueName: \"kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.777811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.777787 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.777903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.777835 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.777903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.777885 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.778021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.777904 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjh92\" (UniqueName: \"kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.778021 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.777935 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.778648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.778626 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.779375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.779357 5367 scope.go:122] "RemoveContainer" containerID="afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75" Oct 29 22:53:22.779676 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:22.779639 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75\": container with ID starting with afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75 not found: ID does not exist" containerID="afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75" Oct 29 22:53:22.779794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.779683 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75"} err="failed to get container status \"afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75\": rpc error: code = NotFound desc = could not find container \"afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75\": container with ID starting with afa313f6fca2e3f777fb839083126efe7f77a5391bec420f65f2d7832b424a75 not found: ID does not exist" Oct 29 22:53:22.780433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.780410 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.800364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.800337 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmf75\" (UniqueName: \"kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75\") pod \"maas-api-revoke-sub-2d0c7143-gxxmm\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.877388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.877310 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:22.878652 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.878630 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.878806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.878788 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.878856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.878840 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.878900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.878869 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.879010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.878996 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.879079 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879062 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.879142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879079 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:22.879142 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879132 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.879261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879134 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.879261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879190 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.879378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879336 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket\") pod \"275ee01a-c16a-42f8-b332-e2728b091f71\" (UID: \"275ee01a-c16a-42f8-b332-e2728b091f71\") " Oct 29 22:53:22.879378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879335 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.879378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879346 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.879658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879633 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjh92\" (UniqueName: \"kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.879658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879653 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.879863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.879744 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.880094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880068 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.880241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880224 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.880328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880244 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.880328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880257 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.880328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880270 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.880328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880285 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275ee01a-c16a-42f8-b332-e2728b091f71-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.880553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.880468 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.881642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.881618 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.882011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.881956 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "kube-api-access-lt8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:22.882737 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.882674 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.882826 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.882808 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:22.882909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.882892 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token" pod "275ee01a-c16a-42f8-b332-e2728b091f71" (UID: "275ee01a-c16a-42f8-b332-e2728b091f71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:22.906624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.906601 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjh92\" (UniqueName: \"kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92\") pod \"maas-api-revoke-sub-1e70fd8e-c68ms\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.979602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.979367 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:22.981658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.981618 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8cn\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-kube-api-access-lt8cn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.981658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.981656 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275ee01a-c16a-42f8-b332-e2728b091f71-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.981821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.981672 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275ee01a-c16a-42f8-b332-e2728b091f71-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.981821 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:22.981685 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275ee01a-c16a-42f8-b332-e2728b091f71-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.131129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.131045 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn"] Oct 29 22:53:23.138746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.138712 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-2tlcn"] Oct 29 22:53:23.199770 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:53:23.199741 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce10230_33d1_4a11_bcc5_4abbcd119011.slice/crio-64296048364ecafbc2c810d42c749dd8ab4c04746f0db3f9220476e7d444f4d0 WatchSource:0}: Error finding container 64296048364ecafbc2c810d42c749dd8ab4c04746f0db3f9220476e7d444f4d0: Status 404 returned error can't find the container with id 64296048364ecafbc2c810d42c749dd8ab4c04746f0db3f9220476e7d444f4d0 Oct 29 22:53:23.201298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.201214 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm"] Oct 29 22:53:23.226315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.226286 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" path="/var/lib/kubelet/pods/275ee01a-c16a-42f8-b332-e2728b091f71/volumes" Oct 29 22:53:23.244419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.244392 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:23.331397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.331370 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms"] Oct 29 22:53:23.356483 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:53:23.356451 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6935b0d_814d_4678_a78c_86f548e3465f.slice/crio-29732e88720b415fc0c30a39a3422177a121e6c6f9068ea8648a205331644849 WatchSource:0}: Error finding container 29732e88720b415fc0c30a39a3422177a121e6c6f9068ea8648a205331644849: Status 404 returned error can't find the container with id 29732e88720b415fc0c30a39a3422177a121e6c6f9068ea8648a205331644849 Oct 29 22:53:23.774415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.774383 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" event={"ID":"a6935b0d-814d-4678-a78c-86f548e3465f","Type":"ContainerStarted","Data":"29732e88720b415fc0c30a39a3422177a121e6c6f9068ea8648a205331644849"} Oct 29 22:53:23.775963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.775941 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" event={"ID":"9ce10230-33d1-4a11-bcc5-4abbcd119011","Type":"ContainerStarted","Data":"7bfd5bbfadcc951aaf47a224769777f3e154a01ffa98600fa4b3bdb23de3a9ad"} Oct 29 22:53:23.776070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.775979 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" event={"ID":"9ce10230-33d1-4a11-bcc5-4abbcd119011","Type":"ContainerStarted","Data":"64296048364ecafbc2c810d42c749dd8ab4c04746f0db3f9220476e7d444f4d0"} Oct 29 22:53:23.818013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:23.817969 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" podStartSLOduration=1.817956022 podStartE2EDuration="1.817956022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:23.814134833 +0000 UTC m=+2270.708209755" watchObservedRunningTime="2026-10-29 22:53:23.817956022 +0000 UTC m=+2270.712030942" Oct 29 22:53:24.788942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:24.788857 5367 generic.go:356] "Generic (PLEG): container finished" podID="a6935b0d-814d-4678-a78c-86f548e3465f" containerID="74f86ffb55d86efc7021779e9379570fd2ac2c53fd8c5bf002dd7b3ec99aa187" exitCode=0 Oct 29 22:53:24.789367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:24.788947 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" event={"ID":"a6935b0d-814d-4678-a78c-86f548e3465f","Type":"ContainerDied","Data":"74f86ffb55d86efc7021779e9379570fd2ac2c53fd8c5bf002dd7b3ec99aa187"} Oct 29 22:53:24.790622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:24.790599 5367 generic.go:356] "Generic (PLEG): container finished" podID="9ce10230-33d1-4a11-bcc5-4abbcd119011" containerID="7bfd5bbfadcc951aaf47a224769777f3e154a01ffa98600fa4b3bdb23de3a9ad" exitCode=0 Oct 29 22:53:24.790770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:24.790680 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" event={"ID":"9ce10230-33d1-4a11-bcc5-4abbcd119011","Type":"ContainerDied","Data":"7bfd5bbfadcc951aaf47a224769777f3e154a01ffa98600fa4b3bdb23de3a9ad"} Oct 29 22:53:24.837449 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:24.837404 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" podStartSLOduration=2.837390381 podStartE2EDuration="2.837390381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:23.857752781 +0000 UTC m=+2270.751827696" watchObservedRunningTime="2026-10-29 22:53:24.837390381 +0000 UTC m=+2271.731465301" Oct 29 22:53:25.984102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:25.984072 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:26.028573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.028547 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:26.032667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.032639 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca\") pod \"9ce10230-33d1-4a11-bcc5-4abbcd119011\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " Oct 29 22:53:26.032828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.032757 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca\") pod \"9ce10230-33d1-4a11-bcc5-4abbcd119011\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " Oct 29 22:53:26.032969 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.032950 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75\" (UniqueName: \"kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75\") pod \"9ce10230-33d1-4a11-bcc5-4abbcd119011\" (UID: \"9ce10230-33d1-4a11-bcc5-4abbcd119011\") " Oct 29 22:53:26.033098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.033071 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca" pod "9ce10230-33d1-4a11-bcc5-4abbcd119011" (UID: "9ce10230-33d1-4a11-bcc5-4abbcd119011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:26.033812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.033782 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:26.035074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.035049 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca" pod "9ce10230-33d1-4a11-bcc5-4abbcd119011" (UID: "9ce10230-33d1-4a11-bcc5-4abbcd119011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:26.035642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.035619 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75" pod "9ce10230-33d1-4a11-bcc5-4abbcd119011" (UID: "9ce10230-33d1-4a11-bcc5-4abbcd119011"). InnerVolumeSpecName "kube-api-access-dmf75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:26.134603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.134569 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca\") pod \"a6935b0d-814d-4678-a78c-86f548e3465f\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " Oct 29 22:53:26.134863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.134742 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92\" (UniqueName: \"kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92\") pod \"a6935b0d-814d-4678-a78c-86f548e3465f\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " Oct 29 22:53:26.134957 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.134878 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca\") pod \"a6935b0d-814d-4678-a78c-86f548e3465f\" (UID: \"a6935b0d-814d-4678-a78c-86f548e3465f\") " Oct 29 22:53:26.135249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.135222 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca" pod "a6935b0d-814d-4678-a78c-86f548e3465f" (UID: "a6935b0d-814d-4678-a78c-86f548e3465f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:26.135523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.135506 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce10230-33d1-4a11-bcc5-4abbcd119011-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:26.135610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.135524 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmf75\" (UniqueName: \"kubernetes.io/projected/9ce10230-33d1-4a11-bcc5-4abbcd119011-kube-api-access-dmf75\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:26.135610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.135535 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:26.136820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.136793 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca" pod "a6935b0d-814d-4678-a78c-86f548e3465f" (UID: "a6935b0d-814d-4678-a78c-86f548e3465f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:26.137008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.136986 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92" pod "a6935b0d-814d-4678-a78c-86f548e3465f" (UID: "a6935b0d-814d-4678-a78c-86f548e3465f"). InnerVolumeSpecName "kube-api-access-rjh92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:26.236570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.236542 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6935b0d-814d-4678-a78c-86f548e3465f-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:26.236570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.236569 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjh92\" (UniqueName: \"kubernetes.io/projected/a6935b0d-814d-4678-a78c-86f548e3465f-kube-api-access-rjh92\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:26.810653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.810621 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" Oct 29 22:53:26.810840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.810653 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms" event={"ID":"a6935b0d-814d-4678-a78c-86f548e3465f","Type":"ContainerDied","Data":"29732e88720b415fc0c30a39a3422177a121e6c6f9068ea8648a205331644849"} Oct 29 22:53:26.810840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.810719 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29732e88720b415fc0c30a39a3422177a121e6c6f9068ea8648a205331644849" Oct 29 22:53:26.812251 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.812225 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" Oct 29 22:53:26.812372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.812265 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm" event={"ID":"9ce10230-33d1-4a11-bcc5-4abbcd119011","Type":"ContainerDied","Data":"64296048364ecafbc2c810d42c749dd8ab4c04746f0db3f9220476e7d444f4d0"} Oct 29 22:53:26.812372 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:26.812304 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64296048364ecafbc2c810d42c749dd8ab4c04746f0db3f9220476e7d444f4d0" Oct 29 22:53:29.656878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:29.656842 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:33.233117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.233080 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87"] Oct 29 22:53:33.243268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.243236 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8ebc60-54r87"] Oct 29 22:53:33.244275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.244243 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:53:33.630752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.630686 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:53:33.687824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.687780 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume\") pod \"64bfe2cc-0407-4452-ac95-a758aec35d52\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " Oct 29 22:53:33.688037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.687874 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c\" (UniqueName: \"kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c\") pod \"64bfe2cc-0407-4452-ac95-a758aec35d52\" (UID: \"64bfe2cc-0407-4452-ac95-a758aec35d52\") " Oct 29 22:53:33.688927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.688504 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume" pod "64bfe2cc-0407-4452-ac95-a758aec35d52" (UID: "64bfe2cc-0407-4452-ac95-a758aec35d52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:33.688927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.688860 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bfe2cc-0407-4452-ac95-a758aec35d52-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:33.690968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.690938 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c" pod "64bfe2cc-0407-4452-ac95-a758aec35d52" (UID: "64bfe2cc-0407-4452-ac95-a758aec35d52"). InnerVolumeSpecName "kube-api-access-pdh7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:33.732448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.732409 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29"] Oct 29 22:53:33.733579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733548 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9ce10230-33d1-4a11-bcc5-4abbcd119011" containerName="revoke-keys" Oct 29 22:53:33.733579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733577 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce10230-33d1-4a11-bcc5-4abbcd119011" containerName="revoke-keys" Oct 29 22:53:33.733804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733637 5367 cpu_manager.go:414] "containerMap: removing container" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" Oct 29 22:53:33.733804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733645 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" Oct 29 22:53:33.733804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733659 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a6935b0d-814d-4678-a78c-86f548e3465f" containerName="revoke-keys" Oct 29 22:53:33.733804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733667 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6935b0d-814d-4678-a78c-86f548e3465f" containerName="revoke-keys" Oct 29 22:53:33.733804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733761 5367 cpu_manager.go:414] "containerMap: removing container" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerName="payload-pre-processing" Oct 29 22:53:33.733804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733771 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerName="payload-pre-processing" Oct 29 22:53:33.734108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733938 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="275ee01a-c16a-42f8-b332-e2728b091f71" containerName="istio-proxy" Oct 29 22:53:33.734108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733951 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6935b0d-814d-4678-a78c-86f548e3465f" containerName="revoke-keys" Oct 29 22:53:33.734108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733972 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerName="payload-pre-processing" Oct 29 22:53:33.734108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.733983 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce10230-33d1-4a11-bcc5-4abbcd119011" containerName="revoke-keys" Oct 29 22:53:33.738747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.738719 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.755610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.755533 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29"] Oct 29 22:53:33.763861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.763830 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:33.764141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.764118 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:53:33.789764 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.789723 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.789945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.789908 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.789945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.789939 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2tvq\" (UniqueName: \"kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.790067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.790024 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdh7c\" (UniqueName: \"kubernetes.io/projected/64bfe2cc-0407-4452-ac95-a758aec35d52-kube-api-access-pdh7c\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:33.891073 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.891041 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2tvq\" (UniqueName: \"kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.891266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.891222 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.891459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.891442 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.891924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.891894 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.893744 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.893723 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.912477 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.912451 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2tvq\" (UniqueName: \"kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq\") pod \"maas-api-revoke-sub-7f927f65-clm29\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:33.917735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.917694 5367 generic.go:356] "Generic (PLEG): container finished" podID="64bfe2cc-0407-4452-ac95-a758aec35d52" containerID="7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04" exitCode=0 Oct 29 22:53:33.917862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.917769 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" event={"ID":"64bfe2cc-0407-4452-ac95-a758aec35d52","Type":"ContainerDied","Data":"7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04"} Oct 29 22:53:33.917862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.917773 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" Oct 29 22:53:33.917862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.917810 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx" event={"ID":"64bfe2cc-0407-4452-ac95-a758aec35d52","Type":"ContainerDied","Data":"9337556ef05d524700aa91860b00c55d48e183bc7e8a4631301335339a00adc0"} Oct 29 22:53:33.917862 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.917830 5367 scope.go:122] "RemoveContainer" containerID="7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04" Oct 29 22:53:33.929627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.929603 5367 scope.go:122] "RemoveContainer" containerID="7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04" Oct 29 22:53:33.929958 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:33.929929 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04\": container with ID starting with 7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04 not found: ID does not exist" containerID="7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04" Oct 29 22:53:33.930062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.929966 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04"} err="failed to get container status \"7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04\": rpc error: code = NotFound desc = could not find container \"7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04\": container with ID starting with 7903044f1396e68d7ee6ab1ecda76f05ef7a86395cd1aae81aa97c3a90c1de04 not found: ID does not exist" Oct 29 22:53:33.976571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.976538 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx"] Oct 29 22:53:33.981325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:33.981300 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dd4c8-b9f49cd6c-wr4nx"] Oct 29 22:53:34.055488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.055463 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:34.380488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.380462 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29"] Oct 29 22:53:34.380858 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:53:34.380476 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ce5d3d_8626_4f65_9f76_ddfe500d2a66.slice/crio-67c1a33545efbd95cb984d5531e7a0fe8b2a765a35d790a0d1a5bd92bbf58e35 WatchSource:0}: Error finding container 67c1a33545efbd95cb984d5531e7a0fe8b2a765a35d790a0d1a5bd92bbf58e35: Status 404 returned error can't find the container with id 67c1a33545efbd95cb984d5531e7a0fe8b2a765a35d790a0d1a5bd92bbf58e35 Oct 29 22:53:34.741166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.741135 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6"] Oct 29 22:53:34.741418 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.741390 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" podUID="b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" containerName="maas-api" containerID="cri-o://e43fb67458754f7e8093c23860be8994867ff382f0f8824d750fe26a7af5c1b3" gracePeriod=30 Oct 29 22:53:34.934982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.934939 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" event={"ID":"76ce5d3d-8626-4f65-9f76-ddfe500d2a66","Type":"ContainerStarted","Data":"0d1e24638524afa6e06af28f0810210979d2e9b7e48538064b20b271c89c8906"} Oct 29 22:53:34.935150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.935003 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" event={"ID":"76ce5d3d-8626-4f65-9f76-ddfe500d2a66","Type":"ContainerStarted","Data":"67c1a33545efbd95cb984d5531e7a0fe8b2a765a35d790a0d1a5bd92bbf58e35"} Oct 29 22:53:34.939427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.939397 5367 generic.go:356] "Generic (PLEG): container finished" podID="b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" containerID="e43fb67458754f7e8093c23860be8994867ff382f0f8824d750fe26a7af5c1b3" exitCode=0 Oct 29 22:53:34.939569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:34.939489 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" event={"ID":"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec","Type":"ContainerDied","Data":"e43fb67458754f7e8093c23860be8994867ff382f0f8824d750fe26a7af5c1b3"} Oct 29 22:53:35.082676 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.082650 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:53:35.111940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.111906 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls\") pod \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " Oct 29 22:53:35.112124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.112033 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca\") pod \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " Oct 29 22:53:35.112318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.112280 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs\") pod \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " Oct 29 22:53:35.112412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.112355 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb\" (UniqueName: \"kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb\") pod \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\" (UID: \"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec\") " Oct 29 22:53:35.112532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.112481 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca" pod "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" (UID: "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:35.113127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.113101 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:35.114417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.114396 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs" pod "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" (UID: "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:35.114607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.114591 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls" pod "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" (UID: "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:35.114677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.114636 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb" pod "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" (UID: "b084b8e7-3416-4e01-a52f-3dbb55a6b0ec"). InnerVolumeSpecName "kube-api-access-cqpmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:35.239722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.234503 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:35.239722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.234542 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpmb\" (UniqueName: \"kubernetes.io/projected/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-kube-api-access-cqpmb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:35.239722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.234557 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:35.361291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.356970 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" podStartSLOduration=2.356945337 podStartE2EDuration="2.356945337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:34.991821118 +0000 UTC m=+2281.885896056" watchObservedRunningTime="2026-10-29 22:53:35.356945337 +0000 UTC m=+2282.251020260" Oct 29 22:53:35.378556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.378517 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bfe2cc-0407-4452-ac95-a758aec35d52" path="/var/lib/kubelet/pods/64bfe2cc-0407-4452-ac95-a758aec35d52/volumes" Oct 29 22:53:35.379264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.379240 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe30707-d7af-4054-8d96-444f13daded1" path="/var/lib/kubelet/pods/cbe30707-d7af-4054-8d96-444f13daded1/volumes" Oct 29 22:53:35.521104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.521067 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q"] Oct 29 22:53:35.532140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.532105 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dd4c8-kgc7q"] Oct 29 22:53:35.825132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.825105 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:53:35.908284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.905456 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn\" (UniqueName: \"kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn\") pod \"98998b4d-6e7e-417c-b4a9-0998313882de\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " Oct 29 22:53:35.908284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.905541 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume\") pod \"98998b4d-6e7e-417c-b4a9-0998313882de\" (UID: \"98998b4d-6e7e-417c-b4a9-0998313882de\") " Oct 29 22:53:35.909182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.908627 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume" pod "98998b4d-6e7e-417c-b4a9-0998313882de" (UID: "98998b4d-6e7e-417c-b4a9-0998313882de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:35.909870 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.909606 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn" pod "98998b4d-6e7e-417c-b4a9-0998313882de" (UID: "98998b4d-6e7e-417c-b4a9-0998313882de"). InnerVolumeSpecName "kube-api-access-rq9zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:35.972741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.972667 5367 generic.go:356] "Generic (PLEG): container finished" podID="98998b4d-6e7e-417c-b4a9-0998313882de" containerID="ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9" exitCode=0 Oct 29 22:53:35.972892 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.972818 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" event={"ID":"98998b4d-6e7e-417c-b4a9-0998313882de","Type":"ContainerDied","Data":"ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9"} Oct 29 22:53:35.972930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.972899 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" event={"ID":"98998b4d-6e7e-417c-b4a9-0998313882de","Type":"ContainerDied","Data":"4fb255555fd778304ed20475e24c090d2b29f0bae26a2148874717bc051ef719"} Oct 29 22:53:35.972992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.972928 5367 scope.go:122] "RemoveContainer" containerID="ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9" Oct 29 22:53:35.973095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.973081 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq" Oct 29 22:53:35.979489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.979462 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" Oct 29 22:53:35.980259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.980229 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6" event={"ID":"b084b8e7-3416-4e01-a52f-3dbb55a6b0ec","Type":"ContainerDied","Data":"f7d5a7168ecadbf02d1564ff0537ec146e73db1f06d52a637b64065bfa23b719"} Oct 29 22:53:35.984215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.984190 5367 generic.go:356] "Generic (PLEG): container finished" podID="76ce5d3d-8626-4f65-9f76-ddfe500d2a66" containerID="0d1e24638524afa6e06af28f0810210979d2e9b7e48538064b20b271c89c8906" exitCode=0 Oct 29 22:53:35.984303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.984251 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" event={"ID":"76ce5d3d-8626-4f65-9f76-ddfe500d2a66","Type":"ContainerDied","Data":"0d1e24638524afa6e06af28f0810210979d2e9b7e48538064b20b271c89c8906"} Oct 29 22:53:35.992220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.992204 5367 scope.go:122] "RemoveContainer" containerID="ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9" Oct 29 22:53:35.993890 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:35.993860 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9\": container with ID starting with ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9 not found: ID does not exist" containerID="ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9" Oct 29 22:53:35.993981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.993903 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9"} err="failed to get container status \"ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9\": rpc error: code = NotFound desc = could not find container \"ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9\": container with ID starting with ff14bf40f4edade5900ae751cfa185f447cb12f0d269269c1c2cdb7d2279c9c9 not found: ID does not exist" Oct 29 22:53:35.993981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:35.993927 5367 scope.go:122] "RemoveContainer" containerID="e43fb67458754f7e8093c23860be8994867ff382f0f8824d750fe26a7af5c1b3" Oct 29 22:53:36.009456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:36.009433 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9zn\" (UniqueName: \"kubernetes.io/projected/98998b4d-6e7e-417c-b4a9-0998313882de-kube-api-access-rq9zn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:36.009570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:36.009459 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98998b4d-6e7e-417c-b4a9-0998313882de-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:36.084228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:36.084181 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6"] Oct 29 22:53:36.087853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:36.087822 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dd4c8-65b4c95b57-qp8w6"] Oct 29 22:53:36.126470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:36.126428 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq"] Oct 29 22:53:36.134298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:36.134270 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8ebc60-78bc6d6bb7-gqdzq"] Oct 29 22:53:37.185311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.185286 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:37.224342 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.224315 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbea13e-a731-475d-b871-41bfb76bae2b" path="/var/lib/kubelet/pods/8fbea13e-a731-475d-b871-41bfb76bae2b/volumes" Oct 29 22:53:37.224860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.224845 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" path="/var/lib/kubelet/pods/98998b4d-6e7e-417c-b4a9-0998313882de/volumes" Oct 29 22:53:37.225328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.225313 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" path="/var/lib/kubelet/pods/b084b8e7-3416-4e01-a52f-3dbb55a6b0ec/volumes" Oct 29 22:53:37.329761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.329738 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca\") pod \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " Oct 29 22:53:37.329872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.329844 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq\" (UniqueName: \"kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq\") pod \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " Oct 29 22:53:37.329994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.329975 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca\") pod \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\" (UID: \"76ce5d3d-8626-4f65-9f76-ddfe500d2a66\") " Oct 29 22:53:37.330092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.330070 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca" pod "76ce5d3d-8626-4f65-9f76-ddfe500d2a66" (UID: "76ce5d3d-8626-4f65-9f76-ddfe500d2a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:37.330796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.330775 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:37.332159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.332128 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq" pod "76ce5d3d-8626-4f65-9f76-ddfe500d2a66" (UID: "76ce5d3d-8626-4f65-9f76-ddfe500d2a66"). InnerVolumeSpecName "kube-api-access-m2tvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:37.332397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.332380 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca" pod "76ce5d3d-8626-4f65-9f76-ddfe500d2a66" (UID: "76ce5d3d-8626-4f65-9f76-ddfe500d2a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:37.433000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.432970 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2tvq\" (UniqueName: \"kubernetes.io/projected/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-kube-api-access-m2tvq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:37.433104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:37.433008 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce5d3d-8626-4f65-9f76-ddfe500d2a66-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:38.006998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:38.006972 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" Oct 29 22:53:38.007159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:38.007007 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29" event={"ID":"76ce5d3d-8626-4f65-9f76-ddfe500d2a66","Type":"ContainerDied","Data":"67c1a33545efbd95cb984d5531e7a0fe8b2a765a35d790a0d1a5bd92bbf58e35"} Oct 29 22:53:38.007159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:38.007061 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67c1a33545efbd95cb984d5531e7a0fe8b2a765a35d790a0d1a5bd92bbf58e35" Oct 29 22:53:40.573795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:40.573757 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624"] Oct 29 22:53:40.574383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:40.574041 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" containerID="cri-o://83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62" gracePeriod=30 Oct 29 22:53:42.572229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:42.572195 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv"] Oct 29 22:53:42.572797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:42.572433 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" containerID="cri-o://a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f" gracePeriod=30 Oct 29 22:53:44.364297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.364266 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2"] Oct 29 22:53:44.365003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.364986 5367 cpu_manager.go:414] "containerMap: removing container" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" containerName="payload-pre-processing" Oct 29 22:53:44.365058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365004 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" containerName="payload-pre-processing" Oct 29 22:53:44.365058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365020 5367 cpu_manager.go:414] "containerMap: removing container" podUID="76ce5d3d-8626-4f65-9f76-ddfe500d2a66" containerName="revoke-keys" Oct 29 22:53:44.365058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365025 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ce5d3d-8626-4f65-9f76-ddfe500d2a66" containerName="revoke-keys" Oct 29 22:53:44.365058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365054 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" containerName="maas-api" Oct 29 22:53:44.365058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365059 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" containerName="maas-api" Oct 29 22:53:44.365225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365178 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="98998b4d-6e7e-417c-b4a9-0998313882de" containerName="payload-pre-processing" Oct 29 22:53:44.365225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365188 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ce5d3d-8626-4f65-9f76-ddfe500d2a66" containerName="revoke-keys" Oct 29 22:53:44.365225 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.365198 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b084b8e7-3416-4e01-a52f-3dbb55a6b0ec" containerName="maas-api" Oct 29 22:53:44.371037 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.371019 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.394353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.394332 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:44.407379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.407349 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:53:44.415456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.415425 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2"] Oct 29 22:53:44.478981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.478951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.479166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.479005 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsr9\" (UniqueName: \"kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.479249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.479223 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.580583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.580544 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.580802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.580624 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsr9\" (UniqueName: \"kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.581132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.580940 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.582199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.582162 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.591494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.591457 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.607871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.607831 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:53:44.608039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.607909 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:44.609879 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.609848 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsr9\" (UniqueName: \"kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9\") pod \"maas-api-revoke-sub-2d5e10ef-bw6p2\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.686093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:44.685906 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:44.998772 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:53:44.998733 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcf36b9_4090_4b5d_a0c6_dbab8dd2aec0.slice/crio-f7aaec5364668166535f816b0ff1ac0b4012a01c0b6aedd1088cdb970c5faa01 WatchSource:0}: Error finding container f7aaec5364668166535f816b0ff1ac0b4012a01c0b6aedd1088cdb970c5faa01: Status 404 returned error can't find the container with id f7aaec5364668166535f816b0ff1ac0b4012a01c0b6aedd1088cdb970c5faa01 Oct 29 22:53:45.001599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:45.001571 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2"] Oct 29 22:53:45.137858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:45.137820 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" event={"ID":"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0","Type":"ContainerStarted","Data":"f7aaec5364668166535f816b0ff1ac0b4012a01c0b6aedd1088cdb970c5faa01"} Oct 29 22:53:45.885272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:45.885212 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:53:46.002203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002167 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.002419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002264 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.002419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002312 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.002419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002372 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.002624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002450 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.002624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002614 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.002760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002630 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:46.002760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.002739 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.003102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.003057 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.003215 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.003165 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo\") pod \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\" (UID: \"e08200b1-1dcf-440c-b1ef-6a66ba84e01a\") " Oct 29 22:53:46.003346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.003317 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.003444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.003334 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.003610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.003590 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.004171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.004148 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.004312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.004299 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.004424 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.004411 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.004535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.004521 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.004636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.004206 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.005516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.005488 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.006350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.006326 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:46.006470 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.006446 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "kube-api-access-2vdfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:46.007603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.007584 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token" pod "e08200b1-1dcf-440c-b1ef-6a66ba84e01a" (UID: "e08200b1-1dcf-440c-b1ef-6a66ba84e01a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:46.106350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.106313 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.106350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.106346 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdfk\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-kube-api-access-2vdfk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.106350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.106356 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.106623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.106367 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.106623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.106377 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08200b1-1dcf-440c-b1ef-6a66ba84e01a-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.150687 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.150591 5367 generic.go:356] "Generic (PLEG): container finished" podID="7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" containerID="847f07b2782344c3813f4614016e09e75c25a0ccc8f87cc20e2d7de0ec2bddaa" exitCode=0 Oct 29 22:53:46.150920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.150678 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" event={"ID":"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0","Type":"ContainerDied","Data":"847f07b2782344c3813f4614016e09e75c25a0ccc8f87cc20e2d7de0ec2bddaa"} Oct 29 22:53:46.152790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.152766 5367 generic.go:356] "Generic (PLEG): container finished" podID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerID="83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62" exitCode=0 Oct 29 22:53:46.152915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.152820 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" Oct 29 22:53:46.152915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.152856 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" event={"ID":"e08200b1-1dcf-440c-b1ef-6a66ba84e01a","Type":"ContainerDied","Data":"83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62"} Oct 29 22:53:46.152915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.152908 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624" event={"ID":"e08200b1-1dcf-440c-b1ef-6a66ba84e01a","Type":"ContainerDied","Data":"2aea0dce0220edd04c40c5c3180a2abfc38a919dbcb05e0b06efea201d3cc442"} Oct 29 22:53:46.153030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.152933 5367 scope.go:122] "RemoveContainer" containerID="83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62" Oct 29 22:53:46.165071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.165047 5367 scope.go:122] "RemoveContainer" containerID="201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99" Oct 29 22:53:46.176076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.176051 5367 scope.go:122] "RemoveContainer" containerID="83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62" Oct 29 22:53:46.176388 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:46.176367 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62\": container with ID starting with 83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62 not found: ID does not exist" containerID="83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62" Oct 29 22:53:46.176474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.176399 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62"} err="failed to get container status \"83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62\": rpc error: code = NotFound desc = could not find container \"83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62\": container with ID starting with 83c7364cb103b855c927767e3c5711e5a5ea285418c573a340c634a692ce1a62 not found: ID does not exist" Oct 29 22:53:46.176474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.176423 5367 scope.go:122] "RemoveContainer" containerID="201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99" Oct 29 22:53:46.176691 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:46.176670 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99\": container with ID starting with 201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99 not found: ID does not exist" containerID="201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99" Oct 29 22:53:46.176755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.176715 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99"} err="failed to get container status \"201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99\": rpc error: code = NotFound desc = could not find container \"201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99\": container with ID starting with 201d48f3ebfe7faaf723bd4eaa7732015d74277edcb2923c2ced1ab9556a4d99 not found: ID does not exist" Oct 29 22:53:46.191914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.191875 5367 patch_prober.go:29] interesting pod/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:53:46.192028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.191927 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:46.262321 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.262289 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624"] Oct 29 22:53:46.271416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:46.270605 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-sw624"] Oct 29 22:53:47.226733 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.226314 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" path="/var/lib/kubelet/pods/e08200b1-1dcf-440c-b1ef-6a66ba84e01a/volumes" Oct 29 22:53:47.332727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.332677 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:47.425462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.425376 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca\") pod \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " Oct 29 22:53:47.425462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.425435 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca\") pod \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " Oct 29 22:53:47.425678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.425603 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9\" (UniqueName: \"kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9\") pod \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\" (UID: \"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0\") " Oct 29 22:53:47.425911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.425877 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca" pod "7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" (UID: "7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:47.426385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.426360 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:47.428268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.428238 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca" pod "7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" (UID: "7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:47.428268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.428255 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9" pod "7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" (UID: "7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0"). InnerVolumeSpecName "kube-api-access-rrsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:47.527442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.527405 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:47.527442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.527436 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrsr9\" (UniqueName: \"kubernetes.io/projected/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0-kube-api-access-rrsr9\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:47.882795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:47.882764 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:53:48.054290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.054248 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.054501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.054336 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.054501 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.054417 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.054635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.054555 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.054635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.054586 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.054808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.054656 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.067078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.055363 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.067078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.055535 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.067078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.055605 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy\") pod \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\" (UID: \"c2441bc5-e772-4c00-aca4-5a6dee3314e7\") " Oct 29 22:53:48.110722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.110011 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:48.110722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.110350 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:48.127614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.123779 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:48.127614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.124025 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:48.147352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.147292 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:48.178721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.159289 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.178721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.159354 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.178721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.159367 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.178721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.159380 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.178721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.159392 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.210592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.206257 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:48.210592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.206688 5367 generic.go:356] "Generic (PLEG): container finished" podID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerID="a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f" exitCode=0 Oct 29 22:53:48.210592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.206836 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" event={"ID":"c2441bc5-e772-4c00-aca4-5a6dee3314e7","Type":"ContainerDied","Data":"a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f"} Oct 29 22:53:48.210592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.206888 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" event={"ID":"c2441bc5-e772-4c00-aca4-5a6dee3314e7","Type":"ContainerDied","Data":"1a44934879704eb1d341386473546fd020a21bb72cd4d4913087c81882f3fbce"} Oct 29 22:53:48.210592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.206957 5367 scope.go:122] "RemoveContainer" containerID="a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f" Oct 29 22:53:48.210592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.207096 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv" Oct 29 22:53:48.211458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.211113 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:48.211923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.211882 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:48.218740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.216534 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4" pod "c2441bc5-e772-4c00-aca4-5a6dee3314e7" (UID: "c2441bc5-e772-4c00-aca4-5a6dee3314e7"). InnerVolumeSpecName "kube-api-access-bljv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:48.218740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.217162 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" event={"ID":"7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0","Type":"ContainerDied","Data":"f7aaec5364668166535f816b0ff1ac0b4012a01c0b6aedd1088cdb970c5faa01"} Oct 29 22:53:48.218740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.217214 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7aaec5364668166535f816b0ff1ac0b4012a01c0b6aedd1088cdb970c5faa01" Oct 29 22:53:48.218740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.217272 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2" Oct 29 22:53:48.232355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.232335 5367 scope.go:122] "RemoveContainer" containerID="e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2" Oct 29 22:53:48.250575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.250546 5367 scope.go:122] "RemoveContainer" containerID="a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f" Oct 29 22:53:48.251445 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:48.251421 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f\": container with ID starting with a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f not found: ID does not exist" containerID="a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f" Oct 29 22:53:48.251535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.251455 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f"} err="failed to get container status \"a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f\": rpc error: code = NotFound desc = could not find container \"a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f\": container with ID starting with a874090544c7760295632d86d3a6a1555b4b9a81a1055361459c785ffd44787f not found: ID does not exist" Oct 29 22:53:48.251535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.251479 5367 scope.go:122] "RemoveContainer" containerID="e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2" Oct 29 22:53:48.251832 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:48.251806 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2\": container with ID starting with e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2 not found: ID does not exist" containerID="e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2" Oct 29 22:53:48.252035 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.252010 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2"} err="failed to get container status \"e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2\": rpc error: code = NotFound desc = could not find container \"e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2\": container with ID starting with e00910f34db2651064a62e6563fda47a3b454b12068fe1d9d50c26b682f58aa2 not found: ID does not exist" Oct 29 22:53:48.261227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.261206 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.261227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.261227 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.261365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.261238 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljv4\" (UniqueName: \"kubernetes.io/projected/c2441bc5-e772-4c00-aca4-5a6dee3314e7-kube-api-access-bljv4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.261365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.261253 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2441bc5-e772-4c00-aca4-5a6dee3314e7-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:53:48.564424 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.564395 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv"] Oct 29 22:53:48.570239 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:48.570213 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8dd4c8-openshift-default-7d77669699-w5dfv"] Oct 29 22:53:49.225610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:53:49.225572 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" path="/var/lib/kubelet/pods/c2441bc5-e772-4c00-aca4-5a6dee3314e7/volumes" Oct 29 22:53:52.332471 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:53:52.332437 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:54:03.074512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:03.074471 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4"] Oct 29 22:54:18.111043 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.110987 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" podUID="d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" containerName="main" containerID="cri-o://455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5" gracePeriod=45 Oct 29 22:54:18.384099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.384073 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:54:18.472265 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472234 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home\") pod \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " Oct 29 22:54:18.472436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht\" (UniqueName: \"kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht\") pod \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " Oct 29 22:54:18.472436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472418 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm\") pod \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " Oct 29 22:54:18.472557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472448 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache\") pod \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " Oct 29 22:54:18.472557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472531 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home" pod "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" (UID: "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:18.472724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472684 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir\") pod \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " Oct 29 22:54:18.472803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472765 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs\") pod \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\" (UID: \"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd\") " Oct 29 22:54:18.472803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472776 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache" pod "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" (UID: "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:18.473011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.472963 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir" pod "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" (UID: "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:18.473498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.473476 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:18.473605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.473502 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:18.473605 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.473517 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:18.475004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.474984 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm" pod "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" (UID: "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:18.475089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.475023 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht" pod "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" (UID: "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd"). InnerVolumeSpecName "kube-api-access-8qcht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:18.475233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.475207 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs" pod "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" (UID: "d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:18.574730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.574681 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:18.574730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.574727 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcht\" (UniqueName: \"kubernetes.io/projected/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-kube-api-access-8qcht\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:18.574730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.574736 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:18.585102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.585069 5367 generic.go:356] "Generic (PLEG): container finished" podID="d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" containerID="455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5" exitCode=0 Oct 29 22:54:18.585218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.585102 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" event={"ID":"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd","Type":"ContainerDied","Data":"455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5"} Oct 29 22:54:18.585218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.585149 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" event={"ID":"d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd","Type":"ContainerDied","Data":"bd4fbc5ef6d5aea208bdc97f78f6444663ca3bd92db30bee6b4ffe883f7c8126"} Oct 29 22:54:18.585218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.585166 5367 scope.go:122] "RemoveContainer" containerID="455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5" Oct 29 22:54:18.585218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.585112 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4" Oct 29 22:54:18.600488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.600461 5367 scope.go:122] "RemoveContainer" containerID="455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5" Oct 29 22:54:18.600799 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:54:18.600775 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5\": container with ID starting with 455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5 not found: ID does not exist" containerID="455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5" Oct 29 22:54:18.600890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.600810 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5"} err="failed to get container status \"455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5\": rpc error: code = NotFound desc = could not find container \"455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5\": container with ID starting with 455365bc7a8d1a68e17bfffe23adaf5ca28825554f6fbf9b1138551ebce4d3d5 not found: ID does not exist" Oct 29 22:54:18.642639 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.642576 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4"] Oct 29 22:54:18.647042 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:18.647019 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/auth-test-model-dbb507-kserve-9598d99cc-ddwv4"] Oct 29 22:54:19.225288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:19.225257 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" path="/var/lib/kubelet/pods/d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd/volumes" Oct 29 22:54:22.340855 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:54:22.340814 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:54:25.035610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.035578 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk"] Oct 29 22:54:25.036276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036261 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" Oct 29 22:54:25.036323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036277 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" Oct 29 22:54:25.036323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036297 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" Oct 29 22:54:25.036323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036302 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" Oct 29 22:54:25.036323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036313 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" containerName="main" Oct 29 22:54:25.036323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036318 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" containerName="main" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036346 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036353 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036367 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" containerName="revoke-keys" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036372 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" containerName="revoke-keys" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036472 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036483 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036489 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b9ea9c-b8b7-4c2b-aec2-a14e941a28fd" containerName="main" Oct 29 22:54:25.036500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036500 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08200b1-1dcf-440c-b1ef-6a66ba84e01a" containerName="istio-proxy" Oct 29 22:54:25.036801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.036508 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" containerName="revoke-keys" Oct 29 22:54:25.041684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.041666 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.059519 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.059488 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk"] Oct 29 22:54:25.067070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.067046 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:54:25.067308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.067279 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:54:25.078307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.078280 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.078524 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.078501 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.078656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.078634 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqvj\" (UniqueName: \"kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.180404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.180371 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.180571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.180431 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqvj\" (UniqueName: \"kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.180571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.180533 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.181127 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.181100 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.182943 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.182922 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.201462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.201437 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqvj\" (UniqueName: \"kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj\") pod \"maas-api-revoke-sub-df93b9bf-lrggk\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.357742 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.355831 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:25.686509 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:54:25.686477 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2248a7_ed0d_404f_b10a_20573d112291.slice/crio-9dfd5f42f56180460dbec58480998b94dd1ac406ccc4a213a29365989b96f3d5 WatchSource:0}: Error finding container 9dfd5f42f56180460dbec58480998b94dd1ac406ccc4a213a29365989b96f3d5: Status 404 returned error can't find the container with id 9dfd5f42f56180460dbec58480998b94dd1ac406ccc4a213a29365989b96f3d5 Oct 29 22:54:25.686691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:25.686665 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk"] Oct 29 22:54:26.660663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:26.660621 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" event={"ID":"3d2248a7-ed0d-404f-b10a-20573d112291","Type":"ContainerStarted","Data":"7e9af4d7fd849c4d11d03e316fe5b271c3215f3fe645f824a577854bea3ba192"} Oct 29 22:54:26.661029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:26.660737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" event={"ID":"3d2248a7-ed0d-404f-b10a-20573d112291","Type":"ContainerStarted","Data":"9dfd5f42f56180460dbec58480998b94dd1ac406ccc4a213a29365989b96f3d5"} Oct 29 22:54:26.705146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:26.705089 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" podStartSLOduration=1.705076335 podStartE2EDuration="1.705076335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:26.700876493 +0000 UTC m=+2333.594951415" watchObservedRunningTime="2026-10-29 22:54:26.705076335 +0000 UTC m=+2333.599151256" Oct 29 22:54:27.670753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:27.670678 5367 generic.go:356] "Generic (PLEG): container finished" podID="3d2248a7-ed0d-404f-b10a-20573d112291" containerID="7e9af4d7fd849c4d11d03e316fe5b271c3215f3fe645f824a577854bea3ba192" exitCode=0 Oct 29 22:54:27.671163 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:27.670776 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" event={"ID":"3d2248a7-ed0d-404f-b10a-20573d112291","Type":"ContainerDied","Data":"7e9af4d7fd849c4d11d03e316fe5b271c3215f3fe645f824a577854bea3ba192"} Oct 29 22:54:28.844878 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.844852 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:28.937939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.937870 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca\") pod \"3d2248a7-ed0d-404f-b10a-20573d112291\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " Oct 29 22:54:28.938093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.938069 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj\" (UniqueName: \"kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj\") pod \"3d2248a7-ed0d-404f-b10a-20573d112291\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " Oct 29 22:54:28.938154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.938138 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca\") pod \"3d2248a7-ed0d-404f-b10a-20573d112291\" (UID: \"3d2248a7-ed0d-404f-b10a-20573d112291\") " Oct 29 22:54:28.938299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.938274 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca" pod "3d2248a7-ed0d-404f-b10a-20573d112291" (UID: "3d2248a7-ed0d-404f-b10a-20573d112291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:28.938684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.938655 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:28.940177 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.940149 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca" pod "3d2248a7-ed0d-404f-b10a-20573d112291" (UID: "3d2248a7-ed0d-404f-b10a-20573d112291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:28.940296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:28.940258 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj" pod "3d2248a7-ed0d-404f-b10a-20573d112291" (UID: "3d2248a7-ed0d-404f-b10a-20573d112291"). InnerVolumeSpecName "kube-api-access-njqvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:29.040486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:29.040454 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njqvj\" (UniqueName: \"kubernetes.io/projected/3d2248a7-ed0d-404f-b10a-20573d112291-kube-api-access-njqvj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:29.040486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:29.040484 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2248a7-ed0d-404f-b10a-20573d112291-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:29.688932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:29.688898 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" Oct 29 22:54:29.689125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:29.688931 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk" event={"ID":"3d2248a7-ed0d-404f-b10a-20573d112291","Type":"ContainerDied","Data":"9dfd5f42f56180460dbec58480998b94dd1ac406ccc4a213a29365989b96f3d5"} Oct 29 22:54:29.689125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:29.688990 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dfd5f42f56180460dbec58480998b94dd1ac406ccc4a213a29365989b96f3d5" Oct 29 22:54:37.494556 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:37.494522 5367 scope.go:122] "RemoveContainer" containerID="37f16fd7427d84bce9f41dbd13250c33143756c70a5c87e334aed96aed10a77b" Oct 29 22:54:37.506379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:37.506353 5367 scope.go:122] "RemoveContainer" containerID="ead33f1907994949fcdef6c1b3e719824fd77f44171b1416208497fe5386ea8f" Oct 29 22:54:37.517675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:37.517655 5367 scope.go:122] "RemoveContainer" containerID="3ea7cbf8c230fc183a6a1c0e84729f1f3f954125e52161a77113f49004142f1f" Oct 29 22:54:39.021751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.021720 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf"] Oct 29 22:54:39.022398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.022383 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3d2248a7-ed0d-404f-b10a-20573d112291" containerName="revoke-keys" Oct 29 22:54:39.022398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.022398 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2248a7-ed0d-404f-b10a-20573d112291" containerName="revoke-keys" Oct 29 22:54:39.022505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.022449 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" Oct 29 22:54:39.022505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.022455 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" Oct 29 22:54:39.022572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.022565 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2441bc5-e772-4c00-aca4-5a6dee3314e7" containerName="istio-proxy" Oct 29 22:54:39.022606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.022579 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2248a7-ed0d-404f-b10a-20573d112291" containerName="revoke-keys" Oct 29 22:54:39.027802 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.027778 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.046221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.046189 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf"] Oct 29 22:54:39.053813 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.053788 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:54:39.053959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.053849 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:54:39.134450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.134420 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.134667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.134482 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25r99\" (UniqueName: \"kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.134667 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.134599 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.248336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.248267 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.248609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.248358 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25r99\" (UniqueName: \"kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.248609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.248466 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.250773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.250745 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.254458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.254430 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.270034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.269996 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25r99\" (UniqueName: \"kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99\") pod \"maas-api-revoke-sub-c633eaa9-5cwwf\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.343484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.343452 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:39.672598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.672575 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf"] Oct 29 22:54:39.675659 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:54:39.675630 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31ddf96_84fc_4d1a_95e9_0f363ad410a9.slice/crio-d46631e2b5ffa5deb187279b093107f0df94d9c339473ad50d18e3e4af0f18f6 WatchSource:0}: Error finding container d46631e2b5ffa5deb187279b093107f0df94d9c339473ad50d18e3e4af0f18f6: Status 404 returned error can't find the container with id d46631e2b5ffa5deb187279b093107f0df94d9c339473ad50d18e3e4af0f18f6 Oct 29 22:54:39.819538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:39.819498 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" event={"ID":"b31ddf96-84fc-4d1a-95e9-0f363ad410a9","Type":"ContainerStarted","Data":"d46631e2b5ffa5deb187279b093107f0df94d9c339473ad50d18e3e4af0f18f6"} Oct 29 22:54:40.832631 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.832591 5367 generic.go:356] "Generic (PLEG): container finished" podID="b31ddf96-84fc-4d1a-95e9-0f363ad410a9" containerID="15e7fdc34d19c5243a0fc1b195b5158dac9fb3d895d5471f387176d3af5f2936" exitCode=0 Oct 29 22:54:40.833148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.832657 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" event={"ID":"b31ddf96-84fc-4d1a-95e9-0f363ad410a9","Type":"ContainerDied","Data":"15e7fdc34d19c5243a0fc1b195b5158dac9fb3d895d5471f387176d3af5f2936"} Oct 29 22:54:40.833521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.833491 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp"] Oct 29 22:54:40.839860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.839836 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.860374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.860347 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-09fc89-openshift-default-dockercfg-tjchp\"" Oct 29 22:54:40.869286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.869262 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp"] Oct 29 22:54:40.975987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.975944 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7x4g\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976067 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976115 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976153 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976181 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976297 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976422 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976503 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:40.976560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:40.976552 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077507 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077553 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077597 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077629 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7x4g\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077734 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077781 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077841 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077884 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077965 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.077992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077977 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.078277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.077995 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.078277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.078259 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.078365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.078264 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.078622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.078602 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.080138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.080120 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.080430 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.080411 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.098931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.098873 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.099972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.099949 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7x4g\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g\") pod \"e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.159264 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.159234 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:41.661918 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:54:41.661886 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b18a7d5_0996_425b_96ce_54196e33acbd.slice/crio-87fad629d60c811c89b0ef783bfdc4ee3cabd583b0986a596abc4ef75fb31dea WatchSource:0}: Error finding container 87fad629d60c811c89b0ef783bfdc4ee3cabd583b0986a596abc4ef75fb31dea: Status 404 returned error can't find the container with id 87fad629d60c811c89b0ef783bfdc4ee3cabd583b0986a596abc4ef75fb31dea Oct 29 22:54:41.662185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.662164 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp"] Oct 29 22:54:41.665380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.665347 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:41.665461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.665413 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:41.665461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.665445 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:41.845416 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.845381 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" event={"ID":"8b18a7d5-0996-425b-96ce-54196e33acbd","Type":"ContainerStarted","Data":"e67e1bbc9330a8b70f06d7491f94b3d651ed7434bbe32dc382f8bd74249c20af"} Oct 29 22:54:41.845902 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.845438 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" event={"ID":"8b18a7d5-0996-425b-96ce-54196e33acbd","Type":"ContainerStarted","Data":"87fad629d60c811c89b0ef783bfdc4ee3cabd583b0986a596abc4ef75fb31dea"} Oct 29 22:54:41.893939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:41.893884 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podStartSLOduration=1.8938692430000001 podStartE2EDuration="1.893869243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:41.889628465 +0000 UTC m=+2348.783703386" watchObservedRunningTime="2026-10-29 22:54:41.893869243 +0000 UTC m=+2348.787944163" Oct 29 22:54:42.065088 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.065063 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:42.146909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.146870 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca\") pod \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " Oct 29 22:54:42.147408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.147116 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca\") pod \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " Oct 29 22:54:42.147625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.147416 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca" pod "b31ddf96-84fc-4d1a-95e9-0f363ad410a9" (UID: "b31ddf96-84fc-4d1a-95e9-0f363ad410a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:42.147625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.147597 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99\" (UniqueName: \"kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99\") pod \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\" (UID: \"b31ddf96-84fc-4d1a-95e9-0f363ad410a9\") " Oct 29 22:54:42.148483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.148434 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:42.149674 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.149650 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca" pod "b31ddf96-84fc-4d1a-95e9-0f363ad410a9" (UID: "b31ddf96-84fc-4d1a-95e9-0f363ad410a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:42.150026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.150006 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99" pod "b31ddf96-84fc-4d1a-95e9-0f363ad410a9" (UID: "b31ddf96-84fc-4d1a-95e9-0f363ad410a9"). InnerVolumeSpecName "kube-api-access-25r99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:42.159521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.159463 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:42.160839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.160814 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 29 22:54:42.160921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.160868 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 29 22:54:42.249624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.249591 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25r99\" (UniqueName: \"kubernetes.io/projected/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-kube-api-access-25r99\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:42.249624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.249624 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31ddf96-84fc-4d1a-95e9-0f363ad410a9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:42.855015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.854984 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" Oct 29 22:54:42.855015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.855005 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf" event={"ID":"b31ddf96-84fc-4d1a-95e9-0f363ad410a9","Type":"ContainerDied","Data":"d46631e2b5ffa5deb187279b093107f0df94d9c339473ad50d18e3e4af0f18f6"} Oct 29 22:54:42.855480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:42.855054 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46631e2b5ffa5deb187279b093107f0df94d9c339473ad50d18e3e4af0f18f6" Oct 29 22:54:43.165199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:43.165125 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:43.938719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:43.937671 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:43.946728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:43.944977 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:54:46.770748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.764376 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b"] Oct 29 22:54:46.770748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.765504 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b31ddf96-84fc-4d1a-95e9-0f363ad410a9" containerName="revoke-keys" Oct 29 22:54:46.770748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.765527 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31ddf96-84fc-4d1a-95e9-0f363ad410a9" containerName="revoke-keys" Oct 29 22:54:46.770748 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.765799 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31ddf96-84fc-4d1a-95e9-0f363ad410a9" containerName="revoke-keys" Oct 29 22:54:46.814726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.798128 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:46.837920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.834916 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09fc89-dockercfg-vqfgb\"" Oct 29 22:54:46.839168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.838133 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b"] Oct 29 22:54:46.848191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.848158 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09fc89\"" Oct 29 22:54:46.902671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.902625 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:46.902924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:46.902800 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjvs\" (UniqueName: \"kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:47.006727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:47.005067 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:47.006727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:47.005207 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjvs\" (UniqueName: \"kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:47.006727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:47.006677 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:47.027561 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:47.027492 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjvs\" (UniqueName: \"kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:47.162918 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:47.162883 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:47.493129 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:54:47.493100 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa40ef16_0cce_4852_bc9d_293db5e7dbd0.slice/crio-066375ee7d89973e241db57874f60b04e32fc460bbae84d94881848a845f807f WatchSource:0}: Error finding container 066375ee7d89973e241db57874f60b04e32fc460bbae84d94881848a845f807f: Status 404 returned error can't find the container with id 066375ee7d89973e241db57874f60b04e32fc460bbae84d94881848a845f807f Oct 29 22:54:47.493592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:47.493569 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b"] Oct 29 22:54:48.014190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:48.014154 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" event={"ID":"aa40ef16-0cce-4852-bc9d-293db5e7dbd0","Type":"ContainerStarted","Data":"8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe"} Oct 29 22:54:48.014755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:48.014214 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" event={"ID":"aa40ef16-0cce-4852-bc9d-293db5e7dbd0","Type":"ContainerStarted","Data":"066375ee7d89973e241db57874f60b04e32fc460bbae84d94881848a845f807f"} Oct 29 22:54:48.014755 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:48.014381 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:54:48.058323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:48.057868 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" podStartSLOduration=2.057849432 podStartE2EDuration="2.057849432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:48.057045845 +0000 UTC m=+2354.951120767" watchObservedRunningTime="2026-10-29 22:54:48.057849432 +0000 UTC m=+2354.951924356" Oct 29 22:54:51.803609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.803574 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44"] Oct 29 22:54:51.808520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.808502 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.829291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.829264 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44"] Oct 29 22:54:51.838741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.836465 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:54:51.838741 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.836531 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:54:51.895493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.895443 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.895662 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.895560 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.895750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.895680 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfv9d\" (UniqueName: \"kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.997237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.997209 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfv9d\" (UniqueName: \"kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.997411 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.997367 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.997460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.997415 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.998072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.998052 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:51.999841 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:51.999816 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:52.016895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:52.016870 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfv9d\" (UniqueName: \"kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d\") pod \"maas-api-revoke-sub-f8918ccd-hqj44\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:52.123482 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:52.123450 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:52.333778 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:54:52.333742 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:54:52.429840 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:54:52.429808 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282335fe_80d1_4c66_8025_837a636e1026.slice/crio-3962b27b4fd99cccf06e657f9353961dfa9979512e93a29215049d0efa27f7ef WatchSource:0}: Error finding container 3962b27b4fd99cccf06e657f9353961dfa9979512e93a29215049d0efa27f7ef: Status 404 returned error can't find the container with id 3962b27b4fd99cccf06e657f9353961dfa9979512e93a29215049d0efa27f7ef Oct 29 22:54:52.430947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:52.430922 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44"] Oct 29 22:54:53.061715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:53.061668 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" event={"ID":"282335fe-80d1-4c66-8025-837a636e1026","Type":"ContainerStarted","Data":"8511e1cb3d7e4d79ad8f5eccb6a54c3066b2919fe01aeef772351b2329194bb5"} Oct 29 22:54:53.062185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:53.061761 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" event={"ID":"282335fe-80d1-4c66-8025-837a636e1026","Type":"ContainerStarted","Data":"3962b27b4fd99cccf06e657f9353961dfa9979512e93a29215049d0efa27f7ef"} Oct 29 22:54:53.104350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:53.104299 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" podStartSLOduration=2.104287295 podStartE2EDuration="2.104287295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:53.101245554 +0000 UTC m=+2359.995320476" watchObservedRunningTime="2026-10-29 22:54:53.104287295 +0000 UTC m=+2359.998362215" Oct 29 22:54:54.071625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.071588 5367 generic.go:356] "Generic (PLEG): container finished" podID="282335fe-80d1-4c66-8025-837a636e1026" containerID="8511e1cb3d7e4d79ad8f5eccb6a54c3066b2919fe01aeef772351b2329194bb5" exitCode=0 Oct 29 22:54:54.072112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.071677 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" event={"ID":"282335fe-80d1-4c66-8025-837a636e1026","Type":"ContainerDied","Data":"8511e1cb3d7e4d79ad8f5eccb6a54c3066b2919fe01aeef772351b2329194bb5"} Oct 29 22:54:54.322860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.322782 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7"] Oct 29 22:54:54.327383 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.327361 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.347235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.347207 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7"] Oct 29 22:54:54.433568 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.433522 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.433823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.433676 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.433823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.433745 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8xj\" (UniqueName: \"kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.535217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.535184 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.535399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.535282 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.535399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.535304 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8xj\" (UniqueName: \"kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.535958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.535930 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.537610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.537588 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.557019 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.556994 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8xj\" (UniqueName: \"kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj\") pod \"maas-api-revoke-sub-1528505d-lrzr7\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.643716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.643627 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:54.947851 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:54:54.947818 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6b166a_67a6_48aa_9c8a_3f744836e3c9.slice/crio-202492cda8f09dd07bdb0aa362e5cb77ed676e262db30fe8afe19c412d06430e WatchSource:0}: Error finding container 202492cda8f09dd07bdb0aa362e5cb77ed676e262db30fe8afe19c412d06430e: Status 404 returned error can't find the container with id 202492cda8f09dd07bdb0aa362e5cb77ed676e262db30fe8afe19c412d06430e Oct 29 22:54:54.948995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:54.948969 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7"] Oct 29 22:54:55.085462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.085424 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" event={"ID":"4e6b166a-67a6-48aa-9c8a-3f744836e3c9","Type":"ContainerStarted","Data":"202492cda8f09dd07bdb0aa362e5cb77ed676e262db30fe8afe19c412d06430e"} Oct 29 22:54:55.228441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.228414 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:55.347591 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.347557 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca\") pod \"282335fe-80d1-4c66-8025-837a636e1026\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " Oct 29 22:54:55.347828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.347613 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d\" (UniqueName: \"kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d\") pod \"282335fe-80d1-4c66-8025-837a636e1026\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " Oct 29 22:54:55.347828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.347749 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca\") pod \"282335fe-80d1-4c66-8025-837a636e1026\" (UID: \"282335fe-80d1-4c66-8025-837a636e1026\") " Oct 29 22:54:55.348066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.348035 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca" pod "282335fe-80d1-4c66-8025-837a636e1026" (UID: "282335fe-80d1-4c66-8025-837a636e1026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:55.349518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.349462 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282335fe-80d1-4c66-8025-837a636e1026-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:55.350512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.350488 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca" pod "282335fe-80d1-4c66-8025-837a636e1026" (UID: "282335fe-80d1-4c66-8025-837a636e1026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:55.350627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.350545 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d" pod "282335fe-80d1-4c66-8025-837a636e1026" (UID: "282335fe-80d1-4c66-8025-837a636e1026"). InnerVolumeSpecName "kube-api-access-vfv9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:55.450577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.450543 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfv9d\" (UniqueName: \"kubernetes.io/projected/282335fe-80d1-4c66-8025-837a636e1026-kube-api-access-vfv9d\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:55.450577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:55.450572 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282335fe-80d1-4c66-8025-837a636e1026-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:56.097388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:56.097347 5367 generic.go:356] "Generic (PLEG): container finished" podID="4e6b166a-67a6-48aa-9c8a-3f744836e3c9" containerID="e5dbec99e5c272eda0389f9178110716682ef3e3b464878152c8f23e72d8ccea" exitCode=0 Oct 29 22:54:56.097812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:56.097430 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" event={"ID":"4e6b166a-67a6-48aa-9c8a-3f744836e3c9","Type":"ContainerDied","Data":"e5dbec99e5c272eda0389f9178110716682ef3e3b464878152c8f23e72d8ccea"} Oct 29 22:54:56.099296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:56.099274 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" Oct 29 22:54:56.099296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:56.099279 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44" event={"ID":"282335fe-80d1-4c66-8025-837a636e1026","Type":"ContainerDied","Data":"3962b27b4fd99cccf06e657f9353961dfa9979512e93a29215049d0efa27f7ef"} Oct 29 22:54:56.099474 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:56.099324 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3962b27b4fd99cccf06e657f9353961dfa9979512e93a29215049d0efa27f7ef" Oct 29 22:54:57.286314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.286286 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:57.381319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.381234 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca\") pod \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " Oct 29 22:54:57.381479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.381393 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj\" (UniqueName: \"kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj\") pod \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " Oct 29 22:54:57.381479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.381456 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca\") pod \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\" (UID: \"4e6b166a-67a6-48aa-9c8a-3f744836e3c9\") " Oct 29 22:54:57.381679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.381648 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca" pod "4e6b166a-67a6-48aa-9c8a-3f744836e3c9" (UID: "4e6b166a-67a6-48aa-9c8a-3f744836e3c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:57.382085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.382063 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:57.383512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.383490 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj" pod "4e6b166a-67a6-48aa-9c8a-3f744836e3c9" (UID: "4e6b166a-67a6-48aa-9c8a-3f744836e3c9"). InnerVolumeSpecName "kube-api-access-ws8xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:57.383625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.383565 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca" pod "4e6b166a-67a6-48aa-9c8a-3f744836e3c9" (UID: "4e6b166a-67a6-48aa-9c8a-3f744836e3c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:57.483311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.483272 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8xj\" (UniqueName: \"kubernetes.io/projected/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-kube-api-access-ws8xj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:57.483311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:57.483306 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6b166a-67a6-48aa-9c8a-3f744836e3c9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:54:58.119006 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:58.118974 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" Oct 29 22:54:58.119192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:58.119004 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7" event={"ID":"4e6b166a-67a6-48aa-9c8a-3f744836e3c9","Type":"ContainerDied","Data":"202492cda8f09dd07bdb0aa362e5cb77ed676e262db30fe8afe19c412d06430e"} Oct 29 22:54:58.119192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:58.119050 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="202492cda8f09dd07bdb0aa362e5cb77ed676e262db30fe8afe19c412d06430e" Oct 29 22:54:59.030594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:54:59.030561 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:55:02.300138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300096 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb"] Oct 29 22:55:02.300814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300799 5367 cpu_manager.go:414] "containerMap: removing container" podUID="282335fe-80d1-4c66-8025-837a636e1026" containerName="revoke-keys" Oct 29 22:55:02.300865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300815 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282335fe-80d1-4c66-8025-837a636e1026" containerName="revoke-keys" Oct 29 22:55:02.300865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300828 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4e6b166a-67a6-48aa-9c8a-3f744836e3c9" containerName="revoke-keys" Oct 29 22:55:02.300865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300834 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6b166a-67a6-48aa-9c8a-3f744836e3c9" containerName="revoke-keys" Oct 29 22:55:02.300987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300977 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="282335fe-80d1-4c66-8025-837a636e1026" containerName="revoke-keys" Oct 29 22:55:02.301024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.300990 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6b166a-67a6-48aa-9c8a-3f744836e3c9" containerName="revoke-keys" Oct 29 22:55:02.304682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.304666 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.325790 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.325760 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb"] Oct 29 22:55:02.332460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.332430 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:02.332621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.332555 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:55:02.456842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.456810 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.456842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.456850 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98j9\" (UniqueName: \"kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.457122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.456873 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.558815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.558779 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.559078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.559056 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98j9\" (UniqueName: \"kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.559241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.559203 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.559974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.559956 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.562066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.562043 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.582050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.582018 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98j9\" (UniqueName: \"kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9\") pod \"maas-api-revoke-sub-4097d25f-tlpdb\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.619313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.619279 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:02.933990 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:02.933952 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24487aa1_d378_419e_ba93_2b5b28013699.slice/crio-6cb265181c748b8e49f6c149b930b816a7177d6a0dd9617e3733ea9d8ff54401 WatchSource:0}: Error finding container 6cb265181c748b8e49f6c149b930b816a7177d6a0dd9617e3733ea9d8ff54401: Status 404 returned error can't find the container with id 6cb265181c748b8e49f6c149b930b816a7177d6a0dd9617e3733ea9d8ff54401 Oct 29 22:55:02.934944 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:02.934923 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb"] Oct 29 22:55:03.163590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:03.163559 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" event={"ID":"24487aa1-d378-419e-ba93-2b5b28013699","Type":"ContainerStarted","Data":"6cb265181c748b8e49f6c149b930b816a7177d6a0dd9617e3733ea9d8ff54401"} Oct 29 22:55:04.174907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:04.174829 5367 generic.go:356] "Generic (PLEG): container finished" podID="24487aa1-d378-419e-ba93-2b5b28013699" containerID="6d0dee3408695d3ee6cac8a32b81407abc4517a4f637a346caaa3326b851f671" exitCode=0 Oct 29 22:55:04.175292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:04.174916 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" event={"ID":"24487aa1-d378-419e-ba93-2b5b28013699","Type":"ContainerDied","Data":"6d0dee3408695d3ee6cac8a32b81407abc4517a4f637a346caaa3326b851f671"} Oct 29 22:55:05.159468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.159425 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr"] Oct 29 22:55:05.163967 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.163947 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.182394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.182355 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr"] Oct 29 22:55:05.233613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.233579 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q"] Oct 29 22:55:05.241199 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.241173 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.250388 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.250359 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q"] Oct 29 22:55:05.322472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.322436 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.322472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.322483 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2xk\" (UniqueName: \"kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.322766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.322589 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.322766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.322629 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.322766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.322741 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqhw\" (UniqueName: \"kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.323053 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.323026 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.423716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.423639 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:05.424214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424192 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.424343 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424331 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.424391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424355 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2xk\" (UniqueName: \"kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.424454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424395 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.424509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424490 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.424562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424539 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqhw\" (UniqueName: \"kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.425018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.424994 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.425248 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.425232 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.426784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.426759 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.426890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.426865 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.445530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.445455 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqhw\" (UniqueName: \"kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw\") pod \"maas-api-revoke-sub-0e49a879-hjj7q\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.446085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.446057 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2xk\" (UniqueName: \"kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk\") pod \"maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.478635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.478601 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:05.525494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.525464 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca\") pod \"24487aa1-d378-419e-ba93-2b5b28013699\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " Oct 29 22:55:05.525677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.525560 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9\" (UniqueName: \"kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9\") pod \"24487aa1-d378-419e-ba93-2b5b28013699\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " Oct 29 22:55:05.525910 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.525870 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca\") pod \"24487aa1-d378-419e-ba93-2b5b28013699\" (UID: \"24487aa1-d378-419e-ba93-2b5b28013699\") " Oct 29 22:55:05.526276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.526243 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca" pod "24487aa1-d378-419e-ba93-2b5b28013699" (UID: "24487aa1-d378-419e-ba93-2b5b28013699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:05.526597 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.526578 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:05.528806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.528782 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca" pod "24487aa1-d378-419e-ba93-2b5b28013699" (UID: "24487aa1-d378-419e-ba93-2b5b28013699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:05.528889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.528811 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9" pod "24487aa1-d378-419e-ba93-2b5b28013699" (UID: "24487aa1-d378-419e-ba93-2b5b28013699"). InnerVolumeSpecName "kube-api-access-g98j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:05.562973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.562942 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:05.628801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.628741 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24487aa1-d378-419e-ba93-2b5b28013699-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:05.628801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.628774 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98j9\" (UniqueName: \"kubernetes.io/projected/24487aa1-d378-419e-ba93-2b5b28013699-kube-api-access-g98j9\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:05.787501 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:05.787470 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5e5606_05fa_43a8_afa9_8c28f5691523.slice/crio-ab67673dca4003df9db47a2bcc112ae2f8d8440e57718bd6e09f8e07ba254ffc WatchSource:0}: Error finding container ab67673dca4003df9db47a2bcc112ae2f8d8440e57718bd6e09f8e07ba254ffc: Status 404 returned error can't find the container with id ab67673dca4003df9db47a2bcc112ae2f8d8440e57718bd6e09f8e07ba254ffc Oct 29 22:55:05.791099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.791072 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr"] Oct 29 22:55:05.883337 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:05.883296 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0827f96b_317f_4abc_b268_3ded5f8cea8d.slice/crio-5db23ed8cdedc9759a18443490868a7fdf36ccf7eaf3ef53ea056cfd42bddcfc WatchSource:0}: Error finding container 5db23ed8cdedc9759a18443490868a7fdf36ccf7eaf3ef53ea056cfd42bddcfc: Status 404 returned error can't find the container with id 5db23ed8cdedc9759a18443490868a7fdf36ccf7eaf3ef53ea056cfd42bddcfc Oct 29 22:55:05.887663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:05.887631 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q"] Oct 29 22:55:06.195547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.195515 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" event={"ID":"ea5e5606-05fa-43a8-afa9-8c28f5691523","Type":"ContainerStarted","Data":"ab67673dca4003df9db47a2bcc112ae2f8d8440e57718bd6e09f8e07ba254ffc"} Oct 29 22:55:06.197237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.197211 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" event={"ID":"0827f96b-317f-4abc-b268-3ded5f8cea8d","Type":"ContainerStarted","Data":"dbbed726f87d84ada14b904404d3b66fa139be4a296b0ae6f8b4903bc3e64185"} Oct 29 22:55:06.197375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.197256 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" event={"ID":"0827f96b-317f-4abc-b268-3ded5f8cea8d","Type":"ContainerStarted","Data":"5db23ed8cdedc9759a18443490868a7fdf36ccf7eaf3ef53ea056cfd42bddcfc"} Oct 29 22:55:06.198654 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.198632 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" Oct 29 22:55:06.198785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.198660 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb" event={"ID":"24487aa1-d378-419e-ba93-2b5b28013699","Type":"ContainerDied","Data":"6cb265181c748b8e49f6c149b930b816a7177d6a0dd9617e3733ea9d8ff54401"} Oct 29 22:55:06.198785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.198714 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cb265181c748b8e49f6c149b930b816a7177d6a0dd9617e3733ea9d8ff54401" Oct 29 22:55:06.262134 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:06.262061 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" podStartSLOduration=1.262046111 podStartE2EDuration="1.262046111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:06.256818771 +0000 UTC m=+2373.150893695" watchObservedRunningTime="2026-10-29 22:55:06.262046111 +0000 UTC m=+2373.156121071" Oct 29 22:55:07.208876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:07.208842 5367 generic.go:356] "Generic (PLEG): container finished" podID="ea5e5606-05fa-43a8-afa9-8c28f5691523" containerID="27e4b2f966264ddf815e96fe7a15616a3900936bf995e7674e3b73c6cf19ff3e" exitCode=0 Oct 29 22:55:07.209364 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:07.208967 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" event={"ID":"ea5e5606-05fa-43a8-afa9-8c28f5691523","Type":"ContainerDied","Data":"27e4b2f966264ddf815e96fe7a15616a3900936bf995e7674e3b73c6cf19ff3e"} Oct 29 22:55:07.210828 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:07.210804 5367 generic.go:356] "Generic (PLEG): container finished" podID="0827f96b-317f-4abc-b268-3ded5f8cea8d" containerID="dbbed726f87d84ada14b904404d3b66fa139be4a296b0ae6f8b4903bc3e64185" exitCode=0 Oct 29 22:55:07.210961 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:07.210887 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" event={"ID":"0827f96b-317f-4abc-b268-3ded5f8cea8d","Type":"ContainerDied","Data":"dbbed726f87d84ada14b904404d3b66fa139be4a296b0ae6f8b4903bc3e64185"} Oct 29 22:55:08.396599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.396571 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:08.441516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.441489 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:08.470139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.470105 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca\") pod \"0827f96b-317f-4abc-b268-3ded5f8cea8d\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " Oct 29 22:55:08.470315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.470234 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw\" (UniqueName: \"kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw\") pod \"0827f96b-317f-4abc-b268-3ded5f8cea8d\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " Oct 29 22:55:08.470315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.470270 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca\") pod \"0827f96b-317f-4abc-b268-3ded5f8cea8d\" (UID: \"0827f96b-317f-4abc-b268-3ded5f8cea8d\") " Oct 29 22:55:08.470508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.470485 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca" pod "0827f96b-317f-4abc-b268-3ded5f8cea8d" (UID: "0827f96b-317f-4abc-b268-3ded5f8cea8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:08.471050 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.470988 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:08.472883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.472857 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw" pod "0827f96b-317f-4abc-b268-3ded5f8cea8d" (UID: "0827f96b-317f-4abc-b268-3ded5f8cea8d"). InnerVolumeSpecName "kube-api-access-hdqhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:08.474547 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.474519 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca" pod "0827f96b-317f-4abc-b268-3ded5f8cea8d" (UID: "0827f96b-317f-4abc-b268-3ded5f8cea8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:08.571874 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.571844 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca\") pod \"ea5e5606-05fa-43a8-afa9-8c28f5691523\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " Oct 29 22:55:08.572075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.572060 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca\") pod \"ea5e5606-05fa-43a8-afa9-8c28f5691523\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " Oct 29 22:55:08.572125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.572097 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk\" (UniqueName: \"kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk\") pod \"ea5e5606-05fa-43a8-afa9-8c28f5691523\" (UID: \"ea5e5606-05fa-43a8-afa9-8c28f5691523\") " Oct 29 22:55:08.572490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.572459 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca" pod "ea5e5606-05fa-43a8-afa9-8c28f5691523" (UID: "ea5e5606-05fa-43a8-afa9-8c28f5691523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:08.572659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.572643 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqhw\" (UniqueName: \"kubernetes.io/projected/0827f96b-317f-4abc-b268-3ded5f8cea8d-kube-api-access-hdqhw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:08.572788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.572662 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0827f96b-317f-4abc-b268-3ded5f8cea8d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:08.572788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.572676 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:08.574074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.574049 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca" pod "ea5e5606-05fa-43a8-afa9-8c28f5691523" (UID: "ea5e5606-05fa-43a8-afa9-8c28f5691523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:08.574172 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.574126 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk" pod "ea5e5606-05fa-43a8-afa9-8c28f5691523" (UID: "ea5e5606-05fa-43a8-afa9-8c28f5691523"). InnerVolumeSpecName "kube-api-access-kg2xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:08.679720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.679069 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2xk\" (UniqueName: \"kubernetes.io/projected/ea5e5606-05fa-43a8-afa9-8c28f5691523-kube-api-access-kg2xk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:08.679720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.679106 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5e5606-05fa-43a8-afa9-8c28f5691523-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:08.876014 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.875980 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq"] Oct 29 22:55:08.876930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.876908 5367 cpu_manager.go:414] "containerMap: removing container" podUID="24487aa1-d378-419e-ba93-2b5b28013699" containerName="revoke-keys" Oct 29 22:55:08.876930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.876928 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24487aa1-d378-419e-ba93-2b5b28013699" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.876950 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0827f96b-317f-4abc-b268-3ded5f8cea8d" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.876957 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0827f96b-317f-4abc-b268-3ded5f8cea8d" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.876967 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ea5e5606-05fa-43a8-afa9-8c28f5691523" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.876973 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5e5606-05fa-43a8-afa9-8c28f5691523" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.877076 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0827f96b-317f-4abc-b268-3ded5f8cea8d" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.877084 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5e5606-05fa-43a8-afa9-8c28f5691523" containerName="revoke-keys" Oct 29 22:55:08.877105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.877099 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="24487aa1-d378-419e-ba93-2b5b28013699" containerName="revoke-keys" Oct 29 22:55:08.881115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.881096 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.906619 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.906454 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq"] Oct 29 22:55:08.961671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.961636 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2ae845-openshift-default-dockercfg-5hvm9\"" Oct 29 22:55:08.986208 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986177 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986315 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986367 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986425 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986460 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986607 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986592 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lq5g\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986640 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986764 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:08.986824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:08.986812 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115164 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lq5g\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115248 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115296 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115348 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115462 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115578 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115605 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115651 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.115710 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.116198 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.116976 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.117215 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.117860 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.120777 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.118103 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.136770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.124093 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.162720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.162145 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lq5g\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.164725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.164164 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.168717 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.168139 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.198721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.197188 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:09.305535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.305501 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" Oct 29 22:55:09.313374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.313348 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" Oct 29 22:55:09.320064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.316529 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q" event={"ID":"0827f96b-317f-4abc-b268-3ded5f8cea8d","Type":"ContainerDied","Data":"5db23ed8cdedc9759a18443490868a7fdf36ccf7eaf3ef53ea056cfd42bddcfc"} Oct 29 22:55:09.320064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.316586 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db23ed8cdedc9759a18443490868a7fdf36ccf7eaf3ef53ea056cfd42bddcfc" Oct 29 22:55:09.320064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.316603 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr" event={"ID":"ea5e5606-05fa-43a8-afa9-8c28f5691523","Type":"ContainerDied","Data":"ab67673dca4003df9db47a2bcc112ae2f8d8440e57718bd6e09f8e07ba254ffc"} Oct 29 22:55:09.320064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.316626 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab67673dca4003df9db47a2bcc112ae2f8d8440e57718bd6e09f8e07ba254ffc" Oct 29 22:55:09.656804 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:09.656772 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28b49d4_923a_418f_96dc_2cca550cc1e5.slice/crio-f63d3eee81980e7ad4270fa0a1d313e8ed835ca4967a382055c2bb6ca4deb5ff WatchSource:0}: Error finding container f63d3eee81980e7ad4270fa0a1d313e8ed835ca4967a382055c2bb6ca4deb5ff: Status 404 returned error can't find the container with id f63d3eee81980e7ad4270fa0a1d313e8ed835ca4967a382055c2bb6ca4deb5ff Oct 29 22:55:09.659740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.659683 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:09.659840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.659783 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:09.659840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.659822 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:09.660858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:09.660833 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq"] Oct 29 22:55:10.323555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:10.323511 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" event={"ID":"c28b49d4-923a-418f-96dc-2cca550cc1e5","Type":"ContainerStarted","Data":"b3b47116b2b94e1c6b7c88fb65b667e4162b88c5bbda5ef9e69839cd5bc34ae4"} Oct 29 22:55:10.323783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:10.323578 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" event={"ID":"c28b49d4-923a-418f-96dc-2cca550cc1e5","Type":"ContainerStarted","Data":"f63d3eee81980e7ad4270fa0a1d313e8ed835ca4967a382055c2bb6ca4deb5ff"} Oct 29 22:55:10.373738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:10.373670 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podStartSLOduration=2.37365715 podStartE2EDuration="2.37365715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:10.368183231 +0000 UTC m=+2377.262258161" watchObservedRunningTime="2026-10-29 22:55:10.37365715 +0000 UTC m=+2377.267732071" Oct 29 22:55:11.198550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:11.198522 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:11.204746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:11.204687 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:11.331780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:11.331744 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:11.332746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:11.332692 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:55:15.608210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:15.608179 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8"] Oct 29 22:55:15.646797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:15.646769 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg"] Oct 29 22:55:16.167234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.167201 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm"] Oct 29 22:55:16.173407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.173386 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.190738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.189298 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:16.190738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.189488 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:55:16.192106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.192081 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm"] Oct 29 22:55:16.209467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.209433 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.209596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.209528 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.209596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.209567 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55pl\" (UniqueName: \"kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.354935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.354078 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.354935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.354150 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.354935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.354191 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55pl\" (UniqueName: \"kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.378719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.370679 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.402181 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.402148 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.482028 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.480625 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55pl\" (UniqueName: \"kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl\") pod \"maas-api-revoke-sub-27de3473-p6bqm\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.491087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.488153 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:16.808612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:16.808565 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm"] Oct 29 22:55:16.814173 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:16.814135 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf12c357_05c8_4ec6_bbea_e74dfca774d3.slice/crio-6d062f5c4a85857f47e09f9dca970c67135900e7e208ec5110cb1b84cf73c102 WatchSource:0}: Error finding container 6d062f5c4a85857f47e09f9dca970c67135900e7e208ec5110cb1b84cf73c102: Status 404 returned error can't find the container with id 6d062f5c4a85857f47e09f9dca970c67135900e7e208ec5110cb1b84cf73c102 Oct 29 22:55:17.490234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.490152 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" event={"ID":"cf12c357-05c8-4ec6-bbea-e74dfca774d3","Type":"ContainerStarted","Data":"dac0a3b76e1a9825bf3f03de3db2dbc16a78390c4b022ae469e7e183a73cb452"} Oct 29 22:55:17.490234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.490204 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" event={"ID":"cf12c357-05c8-4ec6-bbea-e74dfca774d3","Type":"ContainerStarted","Data":"6d062f5c4a85857f47e09f9dca970c67135900e7e208ec5110cb1b84cf73c102"} Oct 29 22:55:17.537306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.537258 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" podStartSLOduration=1.5372452 podStartE2EDuration="1.5372452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:17.533627376 +0000 UTC m=+2384.427702298" watchObservedRunningTime="2026-10-29 22:55:17.5372452 +0000 UTC m=+2384.431320120" Oct 29 22:55:17.554032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.554001 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn"] Oct 29 22:55:17.558890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.558865 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.577684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.577663 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e803b6\"" Oct 29 22:55:17.578391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.578367 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e803b6-dockercfg-6stbh\"" Oct 29 22:55:17.582274 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.582250 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn"] Oct 29 22:55:17.727723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.703861 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bb2b\" (UniqueName: \"kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b\") pod \"payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.727723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.704088 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.797799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.737064 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587"] Oct 29 22:55:17.797799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.751029 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:17.816720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.800515 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587"] Oct 29 22:55:17.816720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.806526 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.816720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.806901 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bb2b\" (UniqueName: \"kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b\") pod \"payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.816720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.808640 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.834856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.830630 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bb2b\" (UniqueName: \"kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b\") pod \"payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.877726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.877175 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:17.908959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.908923 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:17.909160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:17.909134 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2687\" (UniqueName: \"kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687\") pod \"payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.011433 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.011392 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2687\" (UniqueName: \"kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687\") pod \"payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.011776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.011753 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.012495 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.012463 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.033941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.033911 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2687\" (UniqueName: \"kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687\") pod \"payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.083075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.083029 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.197934 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:18.197861 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6ba3b6_78f4_409c_8bfa_aea95ddb30dd.slice/crio-8014e501c98b81548e6e44d35df641bca9933b9988183f6aa7186ffd306bd868 WatchSource:0}: Error finding container 8014e501c98b81548e6e44d35df641bca9933b9988183f6aa7186ffd306bd868: Status 404 returned error can't find the container with id 8014e501c98b81548e6e44d35df641bca9933b9988183f6aa7186ffd306bd868 Oct 29 22:55:18.199789 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.199754 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn"] Oct 29 22:55:18.392618 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:18.392587 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4da448_aa87_450e_b67b_b5bf84787c26.slice/crio-284913f6515bfe27f21e3c77496634eda67658845ea5a236d0007d1632ca8f46 WatchSource:0}: Error finding container 284913f6515bfe27f21e3c77496634eda67658845ea5a236d0007d1632ca8f46: Status 404 returned error can't find the container with id 284913f6515bfe27f21e3c77496634eda67658845ea5a236d0007d1632ca8f46 Oct 29 22:55:18.393734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.393680 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587"] Oct 29 22:55:18.501281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.501245 5367 generic.go:356] "Generic (PLEG): container finished" podID="cf12c357-05c8-4ec6-bbea-e74dfca774d3" containerID="dac0a3b76e1a9825bf3f03de3db2dbc16a78390c4b022ae469e7e183a73cb452" exitCode=0 Oct 29 22:55:18.501437 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.501336 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" event={"ID":"cf12c357-05c8-4ec6-bbea-e74dfca774d3","Type":"ContainerDied","Data":"dac0a3b76e1a9825bf3f03de3db2dbc16a78390c4b022ae469e7e183a73cb452"} Oct 29 22:55:18.503272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.503239 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" event={"ID":"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd","Type":"ContainerStarted","Data":"91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb"} Oct 29 22:55:18.503399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.503306 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:18.503399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.503330 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" event={"ID":"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd","Type":"ContainerStarted","Data":"8014e501c98b81548e6e44d35df641bca9933b9988183f6aa7186ffd306bd868"} Oct 29 22:55:18.504901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.504873 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" event={"ID":"9e4da448-aa87-450e-b67b-b5bf84787c26","Type":"ContainerStarted","Data":"642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407"} Oct 29 22:55:18.504998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.504910 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" event={"ID":"9e4da448-aa87-450e-b67b-b5bf84787c26","Type":"ContainerStarted","Data":"284913f6515bfe27f21e3c77496634eda67658845ea5a236d0007d1632ca8f46"} Oct 29 22:55:18.505062 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.505012 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:18.585269 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.585219 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" podStartSLOduration=1.585202013 podStartE2EDuration="1.585202013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:18.58163952 +0000 UTC m=+2385.475714442" watchObservedRunningTime="2026-10-29 22:55:18.585202013 +0000 UTC m=+2385.479276930" Oct 29 22:55:18.624806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:18.624756 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" podStartSLOduration=1.6247421050000002 podStartE2EDuration="1.624742105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:18.621192046 +0000 UTC m=+2385.515266968" watchObservedRunningTime="2026-10-29 22:55:18.624742105 +0000 UTC m=+2385.518817045" Oct 29 22:55:19.676273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.676248 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:19.839221 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.839187 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca\") pod \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " Oct 29 22:55:19.839405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.839357 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl\" (UniqueName: \"kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl\") pod \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " Oct 29 22:55:19.839522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.839506 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca\") pod \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\" (UID: \"cf12c357-05c8-4ec6-bbea-e74dfca774d3\") " Oct 29 22:55:19.839582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.839519 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca" pod "cf12c357-05c8-4ec6-bbea-e74dfca774d3" (UID: "cf12c357-05c8-4ec6-bbea-e74dfca774d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:19.840195 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.840161 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:19.841939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.841684 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca" pod "cf12c357-05c8-4ec6-bbea-e74dfca774d3" (UID: "cf12c357-05c8-4ec6-bbea-e74dfca774d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:19.841939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.841758 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl" pod "cf12c357-05c8-4ec6-bbea-e74dfca774d3" (UID: "cf12c357-05c8-4ec6-bbea-e74dfca774d3"). InnerVolumeSpecName "kube-api-access-k55pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:19.941847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.941813 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55pl\" (UniqueName: \"kubernetes.io/projected/cf12c357-05c8-4ec6-bbea-e74dfca774d3-kube-api-access-k55pl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:19.941847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:19.941844 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf12c357-05c8-4ec6-bbea-e74dfca774d3-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.524496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:20.524468 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" Oct 29 22:55:20.524820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:20.524489 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm" event={"ID":"cf12c357-05c8-4ec6-bbea-e74dfca774d3","Type":"ContainerDied","Data":"6d062f5c4a85857f47e09f9dca970c67135900e7e208ec5110cb1b84cf73c102"} Oct 29 22:55:20.524820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:20.524545 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d062f5c4a85857f47e09f9dca970c67135900e7e208ec5110cb1b84cf73c102" Oct 29 22:55:20.617564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:20.615799 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerName="payload-processing" containerID="cri-o://63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051" gracePeriod=25 Oct 29 22:55:20.649293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:20.649242 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerName="payload-pre-processing" containerID="cri-o://09f3f20c13ad954973e5818af072fae1d9a395af87759d87569cb4d898a8f8f8" gracePeriod=25 Oct 29 22:55:22.339663 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:22.339620 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:55:26.640707 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.640665 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f"] Oct 29 22:55:26.641439 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.641424 5367 cpu_manager.go:414] "containerMap: removing container" podUID="cf12c357-05c8-4ec6-bbea-e74dfca774d3" containerName="revoke-keys" Oct 29 22:55:26.641504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.641440 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf12c357-05c8-4ec6-bbea-e74dfca774d3" containerName="revoke-keys" Oct 29 22:55:26.641572 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.641563 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf12c357-05c8-4ec6-bbea-e74dfca774d3" containerName="revoke-keys" Oct 29 22:55:26.646564 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.646546 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.666441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.666395 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f"] Oct 29 22:55:26.672465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.672443 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:26.672609 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.672526 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:55:26.816312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.816270 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.816517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.816354 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49wj\" (UniqueName: \"kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.816565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.816552 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.917632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.917557 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.917800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.917739 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.917800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.917766 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49wj\" (UniqueName: \"kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.918352 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.918325 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.919911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.919892 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.938280 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.938248 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49wj\" (UniqueName: \"kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj\") pod \"maas-api-revoke-sub-63407e7b-fd52f\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:26.961162 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:26.961141 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:27.275921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:27.275887 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f"] Oct 29 22:55:27.276071 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:27.275982 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd28b9e5_6491_4e2b_bc1d_29583c298b48.slice/crio-174f432ced4718baf022da81b7b8e6d2dad023b193848f16c3b9a5786225ddf4 WatchSource:0}: Error finding container 174f432ced4718baf022da81b7b8e6d2dad023b193848f16c3b9a5786225ddf4: Status 404 returned error can't find the container with id 174f432ced4718baf022da81b7b8e6d2dad023b193848f16c3b9a5786225ddf4 Oct 29 22:55:27.601988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:27.601954 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" event={"ID":"fd28b9e5-6491-4e2b-bc1d-29583c298b48","Type":"ContainerStarted","Data":"174f432ced4718baf022da81b7b8e6d2dad023b193848f16c3b9a5786225ddf4"} Oct 29 22:55:28.612738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:28.612682 5367 generic.go:356] "Generic (PLEG): container finished" podID="fd28b9e5-6491-4e2b-bc1d-29583c298b48" containerID="59973597d66a79ed5234c1bb35c50b7e3931a9d18a4bff2c0b30b592991e377a" exitCode=0 Oct 29 22:55:28.612738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:28.612725 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" event={"ID":"fd28b9e5-6491-4e2b-bc1d-29583c298b48","Type":"ContainerDied","Data":"59973597d66a79ed5234c1bb35c50b7e3931a9d18a4bff2c0b30b592991e377a"} Oct 29 22:55:29.446091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.446056 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2"] Oct 29 22:55:29.450657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.450641 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.472830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.472803 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2"] Oct 29 22:55:29.480273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.480242 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-09fc89-serving-cert\"" Oct 29 22:55:29.516055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.516025 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 22:55:29.516297 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.516276 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 22:55:29.559341 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.559304 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.559520 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.559428 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.559611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.559548 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkmb\" (UniqueName: \"kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.559611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.559598 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.661238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.661200 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.661791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.661312 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.661791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.661514 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkmb\" (UniqueName: \"kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.661791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.661561 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.662263 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:29.662231 5367 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-serving-cert: secret "maas-api-e2e-ipp-b-09fc89-serving-cert" not found Oct 29 22:55:29.662390 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:29.662308 5367 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls podName:1d36e7b2-183b-473f-bdb6-3a1f38c63efe nodeName:}" failed. No retries permitted until 2026-10-29 22:55:30.16228907 +0000 UTC m=+2397.056363972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls") pod "maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" (UID: "1d36e7b2-183b-473f-bdb6-3a1f38c63efe") : secret "maas-api-e2e-ipp-b-09fc89-serving-cert" not found Oct 29 22:55:29.662743 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.662687 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.665409 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.665364 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.682456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.682428 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkmb\" (UniqueName: \"kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:29.793091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.793065 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:29.974950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.974923 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca\") pod \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " Oct 29 22:55:29.975117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.974973 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca\") pod \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " Oct 29 22:55:29.975117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.975031 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj\" (UniqueName: \"kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj\") pod \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\" (UID: \"fd28b9e5-6491-4e2b-bc1d-29583c298b48\") " Oct 29 22:55:29.975344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.975315 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca" pod "fd28b9e5-6491-4e2b-bc1d-29583c298b48" (UID: "fd28b9e5-6491-4e2b-bc1d-29583c298b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:29.975598 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.975581 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:29.977196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.977176 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj" pod "fd28b9e5-6491-4e2b-bc1d-29583c298b48" (UID: "fd28b9e5-6491-4e2b-bc1d-29583c298b48"). InnerVolumeSpecName "kube-api-access-r49wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:29.977252 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:29.977204 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca" pod "fd28b9e5-6491-4e2b-bc1d-29583c298b48" (UID: "fd28b9e5-6491-4e2b-bc1d-29583c298b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:30.077445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.077413 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd28b9e5-6491-4e2b-bc1d-29583c298b48-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:30.077445 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.077441 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49wj\" (UniqueName: \"kubernetes.io/projected/fd28b9e5-6491-4e2b-bc1d-29583c298b48-kube-api-access-r49wj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:30.179203 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.179170 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:30.181616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.181587 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:30.364988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.364901 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:30.452745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.451067 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:55:30.452745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.451275 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:55:30.638391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.638313 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" Oct 29 22:55:30.638554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.638311 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f" event={"ID":"fd28b9e5-6491-4e2b-bc1d-29583c298b48","Type":"ContainerDied","Data":"174f432ced4718baf022da81b7b8e6d2dad023b193848f16c3b9a5786225ddf4"} Oct 29 22:55:30.638554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.638445 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="174f432ced4718baf022da81b7b8e6d2dad023b193848f16c3b9a5786225ddf4" Oct 29 22:55:30.673713 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:30.673669 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d36e7b2_183b_473f_bdb6_3a1f38c63efe.slice/crio-e53d478231400abef875f4b6f68d113eed120842bcf7b024911caca722c5a4bb WatchSource:0}: Error finding container e53d478231400abef875f4b6f68d113eed120842bcf7b024911caca722c5a4bb: Status 404 returned error can't find the container with id e53d478231400abef875f4b6f68d113eed120842bcf7b024911caca722c5a4bb Oct 29 22:55:30.674911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.674887 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2"] Oct 29 22:55:30.680442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:30.676101 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:55:31.649289 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:31.649250 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" event={"ID":"1d36e7b2-183b-473f-bdb6-3a1f38c63efe","Type":"ContainerStarted","Data":"395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3"} Oct 29 22:55:31.649496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:31.649315 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" event={"ID":"1d36e7b2-183b-473f-bdb6-3a1f38c63efe","Type":"ContainerStarted","Data":"e53d478231400abef875f4b6f68d113eed120842bcf7b024911caca722c5a4bb"} Oct 29 22:55:31.649496 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:31.649344 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:31.696527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:31.696473 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" podStartSLOduration=2.347737789 podStartE2EDuration="2.696458678s" totalImagesPullingTime="348.720889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:31.692348514 +0000 UTC m=+2398.586423437" watchObservedRunningTime="2026-10-29 22:55:31.696458678 +0000 UTC m=+2398.590533608" Oct 29 22:55:37.583672 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:37.583630 5367 scope.go:122] "RemoveContainer" containerID="12d20833c45e3e67b9d2d4ab83d3cfe1d744ef6ec6a145075df6a1208fb9ffe7" Oct 29 22:55:37.666098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:37.666063 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:55:40.450292 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.450250 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:55:40.451288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.451260 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:55:40.820405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.820380 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:55:40.861118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.861077 5367 generic.go:356] "Generic (PLEG): container finished" podID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerID="63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051" exitCode=0 Oct 29 22:55:40.861320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.861203 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" event={"ID":"1d01f693-6f67-46ae-a7fd-0e6f7a48e520","Type":"ContainerDied","Data":"63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051"} Oct 29 22:55:40.861320 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.861294 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" event={"ID":"1d01f693-6f67-46ae-a7fd-0e6f7a48e520","Type":"ContainerDied","Data":"36abc9041346f8a052631b6c88c782057c795b733abd599bade6db640abc0e64"} Oct 29 22:55:40.861463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.861348 5367 scope.go:122] "RemoveContainer" containerID="63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051" Oct 29 22:55:40.861773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.861750 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8" Oct 29 22:55:40.866140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.865870 5367 generic.go:356] "Generic (PLEG): container finished" podID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerID="09f3f20c13ad954973e5818af072fae1d9a395af87759d87569cb4d898a8f8f8" exitCode=0 Oct 29 22:55:40.866140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.866028 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" event={"ID":"454d48ec-548e-4a85-beba-82560fdfb3aa","Type":"ContainerDied","Data":"09f3f20c13ad954973e5818af072fae1d9a395af87759d87569cb4d898a8f8f8"} Oct 29 22:55:40.877831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.877777 5367 scope.go:122] "RemoveContainer" containerID="63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051" Oct 29 22:55:40.878299 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:40.878220 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051\": container with ID starting with 63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051 not found: ID does not exist" containerID="63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051" Oct 29 22:55:40.878299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.878259 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051"} err="failed to get container status \"63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051\": rpc error: code = NotFound desc = could not find container \"63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051\": container with ID starting with 63e10852eacbc0d4912e27406492c73d6df4efbd82087dfdbfebab055f7a0051 not found: ID does not exist" Oct 29 22:55:40.934463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.934431 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:55:40.979673 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.979642 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz\" (UniqueName: \"kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz\") pod \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " Oct 29 22:55:40.979940 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.979923 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume\") pod \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\" (UID: \"1d01f693-6f67-46ae-a7fd-0e6f7a48e520\") " Oct 29 22:55:40.980309 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.980287 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume" pod "1d01f693-6f67-46ae-a7fd-0e6f7a48e520" (UID: "1d01f693-6f67-46ae-a7fd-0e6f7a48e520"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:40.980565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.980547 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:40.982268 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:40.982240 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz" pod "1d01f693-6f67-46ae-a7fd-0e6f7a48e520" (UID: "1d01f693-6f67-46ae-a7fd-0e6f7a48e520"). InnerVolumeSpecName "kube-api-access-qwpfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:41.081814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.081779 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh\" (UniqueName: \"kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh\") pod \"454d48ec-548e-4a85-beba-82560fdfb3aa\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " Oct 29 22:55:41.082008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.081982 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume\") pod \"454d48ec-548e-4a85-beba-82560fdfb3aa\" (UID: \"454d48ec-548e-4a85-beba-82560fdfb3aa\") " Oct 29 22:55:41.082414 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.082380 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume" pod "454d48ec-548e-4a85-beba-82560fdfb3aa" (UID: "454d48ec-548e-4a85-beba-82560fdfb3aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:41.082538 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.082524 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpfz\" (UniqueName: \"kubernetes.io/projected/1d01f693-6f67-46ae-a7fd-0e6f7a48e520-kube-api-access-qwpfz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:41.082635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.082541 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454d48ec-548e-4a85-beba-82560fdfb3aa-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:41.084033 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.084010 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh" pod "454d48ec-548e-4a85-beba-82560fdfb3aa" (UID: "454d48ec-548e-4a85-beba-82560fdfb3aa"). InnerVolumeSpecName "kube-api-access-2qsdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:41.184606 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.184571 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qsdh\" (UniqueName: \"kubernetes.io/projected/454d48ec-548e-4a85-beba-82560fdfb3aa-kube-api-access-2qsdh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:41.213405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.213356 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8"] Oct 29 22:55:41.226658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.226618 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e508-7b86876698-pjzq8"] Oct 29 22:55:41.878486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.878446 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" Oct 29 22:55:41.878486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.878458 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg" event={"ID":"454d48ec-548e-4a85-beba-82560fdfb3aa","Type":"ContainerDied","Data":"4d3b84b29d5dd7c1c691245f3fecbab72af11356afcb39cc5b06cd504bdc9a7f"} Oct 29 22:55:41.879015 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.878538 5367 scope.go:122] "RemoveContainer" containerID="09f3f20c13ad954973e5818af072fae1d9a395af87759d87569cb4d898a8f8f8" Oct 29 22:55:41.936986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.936951 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg"] Oct 29 22:55:41.949635 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:41.949603 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e508-5bcb6b795f-d42tg"] Oct 29 22:55:43.231401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:43.229505 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" path="/var/lib/kubelet/pods/1d01f693-6f67-46ae-a7fd-0e6f7a48e520/volumes" Oct 29 22:55:43.231401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:43.230288 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" path="/var/lib/kubelet/pods/454d48ec-548e-4a85-beba-82560fdfb3aa/volumes" Oct 29 22:55:44.034917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:44.034883 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n"] Oct 29 22:55:44.046179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:44.046147 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f04bf9-njn7n"] Oct 29 22:55:45.026768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.026728 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7"] Oct 29 22:55:45.027237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.026948 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" podUID="575769df-b144-4255-8d0c-bb402c343d43" containerName="maas-api" containerID="cri-o://e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f" gracePeriod=30 Oct 29 22:55:45.236723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.236366 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b523e4d4-01c2-43c1-a454-31bd8734c455" path="/var/lib/kubelet/pods/b523e4d4-01c2-43c1-a454-31bd8734c455/volumes" Oct 29 22:55:45.412725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.410211 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:55:45.526719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.513641 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca\") pod \"575769df-b144-4255-8d0c-bb402c343d43\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " Oct 29 22:55:45.526719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.513826 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\") pod \"575769df-b144-4255-8d0c-bb402c343d43\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " Oct 29 22:55:45.526719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.514002 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs\") pod \"575769df-b144-4255-8d0c-bb402c343d43\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " Oct 29 22:55:45.526719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.514106 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf\" (UniqueName: \"kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf\") pod \"575769df-b144-4255-8d0c-bb402c343d43\" (UID: \"575769df-b144-4255-8d0c-bb402c343d43\") " Oct 29 22:55:45.526719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.518654 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls" pod "575769df-b144-4255-8d0c-bb402c343d43" (UID: "575769df-b144-4255-8d0c-bb402c343d43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:45.526719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.524155 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca" pod "575769df-b144-4255-8d0c-bb402c343d43" (UID: "575769df-b144-4255-8d0c-bb402c343d43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:45.527294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.526818 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs" pod "575769df-b144-4255-8d0c-bb402c343d43" (UID: "575769df-b144-4255-8d0c-bb402c343d43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:45.543227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.533489 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:45.543227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.533515 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575769df-b144-4255-8d0c-bb402c343d43-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:45.543227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.533529 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/575769df-b144-4255-8d0c-bb402c343d43-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:45.544296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.544264 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf" pod "575769df-b144-4255-8d0c-bb402c343d43" (UID: "575769df-b144-4255-8d0c-bb402c343d43"). InnerVolumeSpecName "kube-api-access-n5qgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:45.635380 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.635293 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qgf\" (UniqueName: \"kubernetes.io/projected/575769df-b144-4255-8d0c-bb402c343d43-kube-api-access-n5qgf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:45.921169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.921081 5367 generic.go:356] "Generic (PLEG): container finished" podID="575769df-b144-4255-8d0c-bb402c343d43" containerID="e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f" exitCode=0 Oct 29 22:55:45.921169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.921136 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" event={"ID":"575769df-b144-4255-8d0c-bb402c343d43","Type":"ContainerDied","Data":"e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f"} Oct 29 22:55:45.921385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.921178 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" event={"ID":"575769df-b144-4255-8d0c-bb402c343d43","Type":"ContainerDied","Data":"c154a6a4bb70423a937270c8d91266e262dd99a3be077b790305eceebfb65489"} Oct 29 22:55:45.921385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.921200 5367 scope.go:122] "RemoveContainer" containerID="e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f" Oct 29 22:55:45.921385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.921138 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7" Oct 29 22:55:45.933353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.933332 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr"] Oct 29 22:55:45.934806 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.934765 5367 scope.go:122] "RemoveContainer" containerID="e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f" Oct 29 22:55:45.935276 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:45.935251 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f\": container with ID starting with e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f not found: ID does not exist" containerID="e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f" Oct 29 22:55:45.935360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.935282 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f"} err="failed to get container status \"e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f\": rpc error: code = NotFound desc = could not find container \"e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f\": container with ID starting with e8578893b9c74e6f450b1755e76d0471a47b99317992f1bdfda1ffcdf867cb6f not found: ID does not exist" Oct 29 22:55:45.941324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:45.941298 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e508-7xzdr"] Oct 29 22:55:46.018098 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:46.018000 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7"] Oct 29 22:55:46.053484 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:46.053446 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e508-fd6f944bd-rxlp7"] Oct 29 22:55:47.224592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:47.224555 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575769df-b144-4255-8d0c-bb402c343d43" path="/var/lib/kubelet/pods/575769df-b144-4255-8d0c-bb402c343d43/volumes" Oct 29 22:55:47.225335 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:47.225313 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5e5606-05fa-43a8-afa9-8c28f5691523" path="/var/lib/kubelet/pods/ea5e5606-05fa-43a8-afa9-8c28f5691523/volumes" Oct 29 22:55:49.106261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.106229 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv"] Oct 29 22:55:49.107118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107093 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fd28b9e5-6491-4e2b-bc1d-29583c298b48" containerName="revoke-keys" Oct 29 22:55:49.107118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107117 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd28b9e5-6491-4e2b-bc1d-29583c298b48" containerName="revoke-keys" Oct 29 22:55:49.107299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107151 5367 cpu_manager.go:414] "containerMap: removing container" podUID="575769df-b144-4255-8d0c-bb402c343d43" containerName="maas-api" Oct 29 22:55:49.107299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107160 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575769df-b144-4255-8d0c-bb402c343d43" containerName="maas-api" Oct 29 22:55:49.107299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107185 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerName="payload-processing" Oct 29 22:55:49.107299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107194 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerName="payload-processing" Oct 29 22:55:49.107299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107241 5367 cpu_manager.go:414] "containerMap: removing container" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerName="payload-pre-processing" Oct 29 22:55:49.107299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107249 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerName="payload-pre-processing" Oct 29 22:55:49.107629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107453 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="454d48ec-548e-4a85-beba-82560fdfb3aa" containerName="payload-pre-processing" Oct 29 22:55:49.107629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107476 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd28b9e5-6491-4e2b-bc1d-29583c298b48" containerName="revoke-keys" Oct 29 22:55:49.107629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107488 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="575769df-b144-4255-8d0c-bb402c343d43" containerName="maas-api" Oct 29 22:55:49.107629 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.107501 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d01f693-6f67-46ae-a7fd-0e6f7a48e520" containerName="payload-processing" Oct 29 22:55:49.113535 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.113510 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.133314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.133293 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c6f83c-openshift-default-dockercfg-llt7w\"" Oct 29 22:55:49.155229 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.155194 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv"] Oct 29 22:55:49.197112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197084 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197132 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197256 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197393 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197421 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197443 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197513 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197582 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197567 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.197679 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.197667 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2js\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.302861 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.302914 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.302951 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.302988 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.303028 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303189 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.303168 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2js\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.303228 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.303281 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.303718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.303372 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.304041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.304012 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.304440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.304417 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.313901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.305471 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.314075 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.306032 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.319569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.314759 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.339731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.338417 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2js\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.367766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.359251 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.367766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.359555 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.388722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.387320 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo\") pod \"e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.434720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.434067 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:49.836205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.836159 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv"] Oct 29 22:55:49.840579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.840412 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:49.840651 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.840613 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:49.840740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.840684 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:49.963937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.963898 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" event={"ID":"02720c0f-0336-4df3-8f0c-19853dcd6c59","Type":"ContainerStarted","Data":"fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c"} Oct 29 22:55:49.964125 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:49.963965 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" event={"ID":"02720c0f-0336-4df3-8f0c-19853dcd6c59","Type":"ContainerStarted","Data":"00e2447b4639fa9393d5fdb3968598376f0518a350f2f6c43554f70d9ca8c78b"} Oct 29 22:55:50.015044 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:50.014992 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podStartSLOduration=1.014977363 podStartE2EDuration="1.014977363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:50.010678397 +0000 UTC m=+2416.904753318" watchObservedRunningTime="2026-10-29 22:55:50.014977363 +0000 UTC m=+2416.909052283" Oct 29 22:55:50.435628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:50.435586 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:50.442160 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:50.442137 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:50.972206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:50.972173 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:50.973131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:50.973106 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:55:52.338332 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:52.338291 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:55:53.285907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:53.285864 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8"] Oct 29 22:55:53.286187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:53.286153 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" containerID="cri-o://ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84" gracePeriod=30 Oct 29 22:55:54.063353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.062949 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt"] Oct 29 22:55:54.074186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.074149 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9220efd6-kxkxt"] Oct 29 22:55:54.190997 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.190959 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v"] Oct 29 22:55:54.196123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.196100 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.211796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.211771 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:54.212108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.212088 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:55:54.216132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.216109 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v"] Oct 29 22:55:54.317398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.317310 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.317590 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.317399 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.317715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.317671 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fzs\" (UniqueName: \"kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.419560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.419520 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fzs\" (UniqueName: \"kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.419786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.419674 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.419786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.419710 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.420469 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.420445 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.422081 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.422056 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.440124 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.440089 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fzs\" (UniqueName: \"kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs\") pod \"maas-api-revoke-keys-e2e-inf-b-57d336-r995v\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.512511 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.512471 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:54.828189 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:54.828130 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce0997e_b6e3_47c6_b1af_d462380ca92d.slice/crio-3f1df2d8d3391bad4cf890ae920f1075c797974aaaa41e73e2e25ca615775644 WatchSource:0}: Error finding container 3f1df2d8d3391bad4cf890ae920f1075c797974aaaa41e73e2e25ca615775644: Status 404 returned error can't find the container with id 3f1df2d8d3391bad4cf890ae920f1075c797974aaaa41e73e2e25ca615775644 Oct 29 22:55:54.829315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:54.829291 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v"] Oct 29 22:55:55.008643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.008591 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" event={"ID":"dce0997e-b6e3-47c6-b1af-d462380ca92d","Type":"ContainerStarted","Data":"3f1df2d8d3391bad4cf890ae920f1075c797974aaaa41e73e2e25ca615775644"} Oct 29 22:55:55.190140 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.190056 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw"] Oct 29 22:55:55.196242 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.196217 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.228930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.228896 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b02be5-8992-4244-803a-3a408aae05ae" path="/var/lib/kubelet/pods/c1b02be5-8992-4244-803a-3a408aae05ae/volumes" Oct 29 22:55:55.229394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.229375 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw"] Oct 29 22:55:55.232505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.232480 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.232634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.232549 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8b9p\" (UniqueName: \"kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.267913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.267868 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6f83c\"" Oct 29 22:55:55.321099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.293254 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6f83c-dockercfg-92bfg\"" Oct 29 22:55:55.372793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.350172 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.372793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.350236 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8b9p\" (UniqueName: \"kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.372793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.351480 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.417722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.413652 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8b9p\" (UniqueName: \"kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.484817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.484652 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg"] Oct 29 22:55:55.494010 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.493980 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.511114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.511082 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg"] Oct 29 22:55:55.511381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.511363 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557204 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbpf\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557278 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557307 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557419 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557492 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557565 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557661 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557799 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.559770 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.557843 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.659222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659183 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.659446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659247 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.659446 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659283 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbpf\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.659600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659473 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.659600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659517 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.659865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659832 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659914 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659953 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660054 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.659983 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.660075 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.660192 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.660487 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660636 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.660513 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.660976 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.660950 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.661992 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.661963 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.662310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.662293 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.680786 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.680759 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbpf\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.682119 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.682094 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.814695 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.814661 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:55.853644 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:55.853587 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6c79f8_17d2_4c45_8178_77cb22ab45c6.slice/crio-8ff4d3081e7b6319c50c88eafed9da8d186696c9b7bb05a4a1fc9a57f7e8555a WatchSource:0}: Error finding container 8ff4d3081e7b6319c50c88eafed9da8d186696c9b7bb05a4a1fc9a57f7e8555a: Status 404 returned error can't find the container with id 8ff4d3081e7b6319c50c88eafed9da8d186696c9b7bb05a4a1fc9a57f7e8555a Oct 29 22:55:55.854778 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:55.854745 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw"] Oct 29 22:55:56.019371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.019331 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" event={"ID":"0b6c79f8-17d2-4c45-8178-77cb22ab45c6","Type":"ContainerStarted","Data":"0e29a4b17ca838fc86354d63acae1b47e07ab4aaed918a1217630a511c72d7dc"} Oct 29 22:55:56.019545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.019400 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" event={"ID":"0b6c79f8-17d2-4c45-8178-77cb22ab45c6","Type":"ContainerStarted","Data":"8ff4d3081e7b6319c50c88eafed9da8d186696c9b7bb05a4a1fc9a57f7e8555a"} Oct 29 22:55:56.019545 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.019431 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:55:56.021754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.021724 5367 generic.go:356] "Generic (PLEG): container finished" podID="dce0997e-b6e3-47c6-b1af-d462380ca92d" containerID="a9685943ddbc9eff43d0986a57a9cb38370102e8f56c9370c4d042fead1729f6" exitCode=0 Oct 29 22:55:56.021903 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.021777 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" event={"ID":"dce0997e-b6e3-47c6-b1af-d462380ca92d","Type":"ContainerDied","Data":"a9685943ddbc9eff43d0986a57a9cb38370102e8f56c9370c4d042fead1729f6"} Oct 29 22:55:56.069663 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.069571 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" podStartSLOduration=1.069557261 podStartE2EDuration="1.069557261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:56.064527997 +0000 UTC m=+2422.958602919" watchObservedRunningTime="2026-10-29 22:55:56.069557261 +0000 UTC m=+2422.963632181" Oct 29 22:55:56.128481 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:55:56.128445 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b4153f_06e8_43c1_9121_4cdbc45bb6b7.slice/crio-6a737d8a3b0bf7eb2e7c2a7dd1720b16f39bc26099f50d52519a7b369798b929 WatchSource:0}: Error finding container 6a737d8a3b0bf7eb2e7c2a7dd1720b16f39bc26099f50d52519a7b369798b929: Status 404 returned error can't find the container with id 6a737d8a3b0bf7eb2e7c2a7dd1720b16f39bc26099f50d52519a7b369798b929 Oct 29 22:55:56.128876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.128856 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg"] Oct 29 22:55:56.131326 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.131288 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:56.131448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.131365 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:56.131513 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:56.131413 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:57.033129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.033085 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" event={"ID":"03b4153f-06e8-43c1-9121-4cdbc45bb6b7","Type":"ContainerStarted","Data":"832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1"} Oct 29 22:55:57.033647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.033156 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" event={"ID":"03b4153f-06e8-43c1-9121-4cdbc45bb6b7","Type":"ContainerStarted","Data":"6a737d8a3b0bf7eb2e7c2a7dd1720b16f39bc26099f50d52519a7b369798b929"} Oct 29 22:55:57.078078 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.076835 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podStartSLOduration=2.076816469 podStartE2EDuration="2.076816469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.075825093 +0000 UTC m=+2423.969900028" watchObservedRunningTime="2026-10-29 22:55:57.076816469 +0000 UTC m=+2423.970891391" Oct 29 22:55:57.231000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.230972 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:57.289939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.289861 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca\") pod \"dce0997e-b6e3-47c6-b1af-d462380ca92d\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " Oct 29 22:55:57.289939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.289921 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca\") pod \"dce0997e-b6e3-47c6-b1af-d462380ca92d\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " Oct 29 22:55:57.290180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.290105 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs\" (UniqueName: \"kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs\") pod \"dce0997e-b6e3-47c6-b1af-d462380ca92d\" (UID: \"dce0997e-b6e3-47c6-b1af-d462380ca92d\") " Oct 29 22:55:57.290298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.290266 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca" pod "dce0997e-b6e3-47c6-b1af-d462380ca92d" (UID: "dce0997e-b6e3-47c6-b1af-d462380ca92d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:57.291007 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.290978 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:57.292406 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.292379 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs" pod "dce0997e-b6e3-47c6-b1af-d462380ca92d" (UID: "dce0997e-b6e3-47c6-b1af-d462380ca92d"). InnerVolumeSpecName "kube-api-access-p6fzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:57.292500 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.292485 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca" pod "dce0997e-b6e3-47c6-b1af-d462380ca92d" (UID: "dce0997e-b6e3-47c6-b1af-d462380ca92d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:57.392459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.392425 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fzs\" (UniqueName: \"kubernetes.io/projected/dce0997e-b6e3-47c6-b1af-d462380ca92d-kube-api-access-p6fzs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:57.392459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.392457 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce0997e-b6e3-47c6-b1af-d462380ca92d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:57.841722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.841289 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:55:57.858723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.849236 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.858723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:57.849310 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 29 22:55:58.044543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.044505 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" Oct 29 22:55:58.044543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.044529 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v" event={"ID":"dce0997e-b6e3-47c6-b1af-d462380ca92d","Type":"ContainerDied","Data":"3f1df2d8d3391bad4cf890ae920f1075c797974aaaa41e73e2e25ca615775644"} Oct 29 22:55:58.045018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.044576 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f1df2d8d3391bad4cf890ae920f1075c797974aaaa41e73e2e25ca615775644" Oct 29 22:55:58.637332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.637298 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:55:58.732583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732547 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.732832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732617 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.732832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732659 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.732832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732730 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.732832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732798 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.732832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732825 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.733106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732890 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.733106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.732959 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.733106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.733095 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket\") pod \"7eea7308-86cf-448e-ae51-090ff57081ff\" (UID: \"7eea7308-86cf-448e-ae51-090ff57081ff\") " Oct 29 22:55:58.733571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.733536 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.733948 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.733914 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.734795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.734752 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.734795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.734783 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.735358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.735329 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.735527 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.735505 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:58.735587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.735569 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.737728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.737683 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:58.738131 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.738103 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:55:58.738839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.738816 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.739212 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.739181 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc" pod "7eea7308-86cf-448e-ae51-090ff57081ff" (UID: "7eea7308-86cf-448e-ae51-090ff57081ff"). InnerVolumeSpecName "kube-api-access-nwnsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:58.815831 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.815797 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.815982 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.815849 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 29 22:55:58.835890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835858 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eea7308-86cf-448e-ae51-090ff57081ff-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.835890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835884 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.835890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835893 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.836117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835904 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eea7308-86cf-448e-ae51-090ff57081ff-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.836117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835913 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.836117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835921 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwnsc\" (UniqueName: \"kubernetes.io/projected/7eea7308-86cf-448e-ae51-090ff57081ff-kube-api-access-nwnsc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.836117 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:58.835930 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eea7308-86cf-448e-ae51-090ff57081ff-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:55:59.084067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.084024 5367 generic.go:356] "Generic (PLEG): container finished" podID="7eea7308-86cf-448e-ae51-090ff57081ff" containerID="ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84" exitCode=0 Oct 29 22:55:59.085080 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.085059 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" Oct 29 22:55:59.085291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.085262 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" event={"ID":"7eea7308-86cf-448e-ae51-090ff57081ff","Type":"ContainerDied","Data":"ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84"} Oct 29 22:55:59.085392 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.085366 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8" event={"ID":"7eea7308-86cf-448e-ae51-090ff57081ff","Type":"ContainerDied","Data":"cd234faf4b1b39f55295e57907934c3da88226cc9802df6601022c91525f621d"} Oct 29 22:55:59.085461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.085401 5367 scope.go:122] "RemoveContainer" containerID="ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84" Oct 29 22:55:59.099809 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.099789 5367 scope.go:122] "RemoveContainer" containerID="85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea" Oct 29 22:55:59.111803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.111784 5367 scope.go:122] "RemoveContainer" containerID="ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84" Oct 29 22:55:59.112154 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:59.112134 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84\": container with ID starting with ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84 not found: ID does not exist" containerID="ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84" Oct 29 22:55:59.112266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.112167 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84"} err="failed to get container status \"ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84\": rpc error: code = NotFound desc = could not find container \"ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84\": container with ID starting with ef51e968153827e65d20a09a9b275e22ead170501b23d4ace0aeb691244aad84 not found: ID does not exist" Oct 29 22:55:59.112266 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.112187 5367 scope.go:122] "RemoveContainer" containerID="85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea" Oct 29 22:55:59.112525 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:55:59.112490 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea\": container with ID starting with 85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea not found: ID does not exist" containerID="85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea" Oct 29 22:55:59.112596 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.112535 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea"} err="failed to get container status \"85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea\": rpc error: code = NotFound desc = could not find container \"85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea\": container with ID starting with 85c0dd23aaf5a9a477f30e568f2ca2cd0c5f4dc4b2599df67d93e0f9b5ccf5ea not found: ID does not exist" Oct 29 22:55:59.144337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.144308 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8"] Oct 29 22:55:59.149353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.149324 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-z4cm8"] Oct 29 22:55:59.224450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.224415 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" path="/var/lib/kubelet/pods/7eea7308-86cf-448e-ae51-090ff57081ff/volumes" Oct 29 22:55:59.338346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.338256 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt"] Oct 29 22:55:59.815911 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.815872 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 29 22:55:59.816104 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:55:59.815927 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 29 22:56:00.451824 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.451790 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt"] Oct 29 22:56:00.452539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452524 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" Oct 29 22:56:00.452580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452540 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" Oct 29 22:56:00.452615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452582 5367 cpu_manager.go:414] "containerMap: removing container" podUID="dce0997e-b6e3-47c6-b1af-d462380ca92d" containerName="revoke-keys" Oct 29 22:56:00.452615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452588 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce0997e-b6e3-47c6-b1af-d462380ca92d" containerName="revoke-keys" Oct 29 22:56:00.452719 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452693 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce0997e-b6e3-47c6-b1af-d462380ca92d" containerName="revoke-keys" Oct 29 22:56:00.452766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452728 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" Oct 29 22:56:00.452766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.452742 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" Oct 29 22:56:00.460732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.458726 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.512354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.512325 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt"] Oct 29 22:56:00.515630 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.515603 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1dabb8-openshift-default-dockercfg-skwhs\"" Oct 29 22:56:00.611107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611061 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611123 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611185 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611219 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611304 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhrv\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611375 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611428 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611532 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.611766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.611562 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.723782 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.723840 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.723896 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.723931 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.724016 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhrv\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.724081 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.724135 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.724228 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.724258 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.724771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.724752 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.725419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.725032 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.725419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.725248 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.737800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.725757 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.737800 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.726416 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.750623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.748660 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.762727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.760756 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.762727 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.761515 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhrv\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.801721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.781473 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.818691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.817108 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:00.818691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.818006 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 29 22:56:00.818691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:00.818059 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 29 22:56:01.349731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:01.341561 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:01.350190 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:01.350151 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:01.350387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:01.350355 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:01.351099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:01.347306 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt"] Oct 29 22:56:01.815714 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:01.815674 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.816204 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:01.815773 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 29 22:56:02.035379 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.035334 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp"] Oct 29 22:56:02.048941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.048905 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee831daa-bdnsp"] Oct 29 22:56:02.121849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.121765 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" event={"ID":"0229cbbd-cd97-4074-9d0c-b07aaf7063ba","Type":"ContainerStarted","Data":"a6ffe6d5198cba2c077811b86643645434dad68d4c6c649de56bbfccaffe1096"} Oct 29 22:56:02.121849 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.121838 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" event={"ID":"0229cbbd-cd97-4074-9d0c-b07aaf7063ba","Type":"ContainerStarted","Data":"31b6fb056069bf040cf6d14de791a46a8ece2f22c1e5bd5c70857b68d9b78253"} Oct 29 22:56:02.183771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.175175 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podStartSLOduration=2.175153893 podStartE2EDuration="2.175153893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:02.168099411 +0000 UTC m=+2429.062174334" watchObservedRunningTime="2026-10-29 22:56:02.175153893 +0000 UTC m=+2429.069228815" Oct 29 22:56:02.815456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.815413 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.815634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.815487 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 29 22:56:02.817566 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.817540 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:02.823872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:02.823843 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:03.131623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:03.131528 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:03.132658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:03.132628 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:56:03.236169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:03.236125 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfcc43c-ffa5-491d-9a41-69ce0c869823" path="/var/lib/kubelet/pods/7dfcc43c-ffa5-491d-9a41-69ce0c869823/volumes" Oct 29 22:56:03.820472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:03.820437 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:56:03.820947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:03.820826 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:56:03.821594 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:03.821577 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 22:56:06.918159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.918119 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v"] Oct 29 22:56:06.918964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.918947 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" Oct 29 22:56:06.919022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.918965 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea7308-86cf-448e-ae51-090ff57081ff" containerName="istio-proxy" Oct 29 22:56:06.924479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.924456 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:06.945095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.945061 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v"] Oct 29 22:56:06.950792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.950671 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1dabb8\"" Oct 29 22:56:06.952830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:06.950878 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1dabb8-dockercfg-csfc7\"" Oct 29 22:56:07.077730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.051111 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.224413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.224312 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbxh\" (UniqueName: \"kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.228417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.193532 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:56:07.228417 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.228188 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9"] Oct 29 22:56:07.256887 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.256855 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.273107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.273072 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9"] Oct 29 22:56:07.331491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.331450 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.331671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.331564 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn2l\" (UniqueName: \"kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l\") pod \"payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.331797 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.331754 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.332064 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.332043 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbxh\" (UniqueName: \"kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.332485 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.332466 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.354488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.354452 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbxh\" (UniqueName: \"kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.433691 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.433648 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn2l\" (UniqueName: \"kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l\") pod \"payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.434049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.434027 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.434784 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.434761 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.462740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.462664 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn2l\" (UniqueName: \"kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l\") pod \"payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.550492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.550383 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:07.575405 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.575373 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:07.899775 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:56:07.899741 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80da96b_fb5c_4231_9d06_f65a095081be.slice/crio-b264a4c013cd8a89cfecc0dec2eb5513fd555fd99044cd320aeb5f2bac21f9f1 WatchSource:0}: Error finding container b264a4c013cd8a89cfecc0dec2eb5513fd555fd99044cd320aeb5f2bac21f9f1: Status 404 returned error can't find the container with id b264a4c013cd8a89cfecc0dec2eb5513fd555fd99044cd320aeb5f2bac21f9f1 Oct 29 22:56:07.900659 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.900637 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v"] Oct 29 22:56:07.960195 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:56:07.960162 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04930aae_6fe9_4061_9676_470076591f5e.slice/crio-c7b62cf37536843120dfb68b7f1310e886ea3dc2bfccd41edc94c53562ab90da WatchSource:0}: Error finding container c7b62cf37536843120dfb68b7f1310e886ea3dc2bfccd41edc94c53562ab90da: Status 404 returned error can't find the container with id c7b62cf37536843120dfb68b7f1310e886ea3dc2bfccd41edc94c53562ab90da Oct 29 22:56:07.965327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:07.965299 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9"] Oct 29 22:56:08.272465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.272349 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" event={"ID":"04930aae-6fe9-4061-9676-470076591f5e","Type":"ContainerStarted","Data":"3edf5c46d10f77602ba3c819eddaf712380baafe105e0f021f228b2b9aa3b9c8"} Oct 29 22:56:08.272465 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.272414 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" event={"ID":"04930aae-6fe9-4061-9676-470076591f5e","Type":"ContainerStarted","Data":"c7b62cf37536843120dfb68b7f1310e886ea3dc2bfccd41edc94c53562ab90da"} Oct 29 22:56:08.272729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.272463 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:08.274143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.274113 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" event={"ID":"a80da96b-fb5c-4231-9d06-f65a095081be","Type":"ContainerStarted","Data":"846a276635c8b11bde523c20f9353ce9a9570c8eb4cade445f16d5fa5cf40c8b"} Oct 29 22:56:08.274261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.274166 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:08.274261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.274182 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" event={"ID":"a80da96b-fb5c-4231-9d06-f65a095081be","Type":"ContainerStarted","Data":"b264a4c013cd8a89cfecc0dec2eb5513fd555fd99044cd320aeb5f2bac21f9f1"} Oct 29 22:56:08.318108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.318054 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" podStartSLOduration=2.318039279 podStartE2EDuration="2.318039279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:08.315100272 +0000 UTC m=+2435.209175194" watchObservedRunningTime="2026-10-29 22:56:08.318039279 +0000 UTC m=+2435.212114193" Oct 29 22:56:08.361972 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:08.361920 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" podStartSLOduration=2.361904706 podStartE2EDuration="2.361904706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:08.357235366 +0000 UTC m=+2435.251310291" watchObservedRunningTime="2026-10-29 22:56:08.361904706 +0000 UTC m=+2435.255979665" Oct 29 22:56:14.385187 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.385088 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" podUID="5649e738-d647-4ac7-a618-375f8aa91fc2" containerName="main" containerID="cri-o://c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb" gracePeriod=45 Oct 29 22:56:14.717927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.717895 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:56:14.820077 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820035 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir\") pod \"5649e738-d647-4ac7-a618-375f8aa91fc2\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " Oct 29 22:56:14.820263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820244 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs\") pod \"5649e738-d647-4ac7-a618-375f8aa91fc2\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " Oct 29 22:56:14.820367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820332 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home\") pod \"5649e738-d647-4ac7-a618-375f8aa91fc2\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " Oct 29 22:56:14.820367 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820333 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir" pod "5649e738-d647-4ac7-a618-375f8aa91fc2" (UID: "5649e738-d647-4ac7-a618-375f8aa91fc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:14.820552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820534 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache\") pod \"5649e738-d647-4ac7-a618-375f8aa91fc2\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " Oct 29 22:56:14.820618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820588 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp\" (UniqueName: \"kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp\") pod \"5649e738-d647-4ac7-a618-375f8aa91fc2\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " Oct 29 22:56:14.820683 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820640 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm\") pod \"5649e738-d647-4ac7-a618-375f8aa91fc2\" (UID: \"5649e738-d647-4ac7-a618-375f8aa91fc2\") " Oct 29 22:56:14.821103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.820859 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home" pod "5649e738-d647-4ac7-a618-375f8aa91fc2" (UID: "5649e738-d647-4ac7-a618-375f8aa91fc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:14.821415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.821364 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:14.821415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.821390 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:14.821773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.821749 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache" pod "5649e738-d647-4ac7-a618-375f8aa91fc2" (UID: "5649e738-d647-4ac7-a618-375f8aa91fc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:14.823426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.823403 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm" pod "5649e738-d647-4ac7-a618-375f8aa91fc2" (UID: "5649e738-d647-4ac7-a618-375f8aa91fc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:14.823516 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.823494 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs" pod "5649e738-d647-4ac7-a618-375f8aa91fc2" (UID: "5649e738-d647-4ac7-a618-375f8aa91fc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:14.823585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.823557 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp" pod "5649e738-d647-4ac7-a618-375f8aa91fc2" (UID: "5649e738-d647-4ac7-a618-375f8aa91fc2"). InnerVolumeSpecName "kube-api-access-fkrdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:14.922599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.922559 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5649e738-d647-4ac7-a618-375f8aa91fc2-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:14.922599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.922593 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:14.922599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.922603 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkrdp\" (UniqueName: \"kubernetes.io/projected/5649e738-d647-4ac7-a618-375f8aa91fc2-kube-api-access-fkrdp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:14.922901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:14.922614 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5649e738-d647-4ac7-a618-375f8aa91fc2-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:15.339927 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.339895 5367 generic.go:356] "Generic (PLEG): container finished" podID="5649e738-d647-4ac7-a618-375f8aa91fc2" containerID="c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb" exitCode=0 Oct 29 22:56:15.340091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.339950 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" Oct 29 22:56:15.340091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.339990 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" event={"ID":"5649e738-d647-4ac7-a618-375f8aa91fc2","Type":"ContainerDied","Data":"c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb"} Oct 29 22:56:15.340091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.340038 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt" event={"ID":"5649e738-d647-4ac7-a618-375f8aa91fc2","Type":"ContainerDied","Data":"dfa63fa961d54ed7a0e6464625ff0284c1d9233c20fdea1d2a663c93f654e7f8"} Oct 29 22:56:15.340091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.340061 5367 scope.go:122] "RemoveContainer" containerID="c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb" Oct 29 22:56:15.351987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.351967 5367 scope.go:122] "RemoveContainer" containerID="c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb" Oct 29 22:56:15.352305 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:56:15.352281 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb\": container with ID starting with c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb not found: ID does not exist" containerID="c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb" Oct 29 22:56:15.352394 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.352318 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb"} err="failed to get container status \"c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb\": rpc error: code = NotFound desc = could not find container \"c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb\": container with ID starting with c2ca64d9da826baeb2891caa070bc7c51955adce1403b89275f9d35a8b8353fb not found: ID does not exist" Oct 29 22:56:15.405807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.405774 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt"] Oct 29 22:56:15.410695 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:15.410669 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-6bb06a/sub-test-model-6bb06a-kserve-6847db6fcf-r8bnt"] Oct 29 22:56:17.224369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:17.224334 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5649e738-d647-4ac7-a618-375f8aa91fc2" path="/var/lib/kubelet/pods/5649e738-d647-4ac7-a618-375f8aa91fc2/volumes" Oct 29 22:56:19.286415 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:19.286323 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:56:19.286868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:19.286524 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:56:21.532858 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:21.532813 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b"] Oct 29 22:56:22.336610 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:56:22.336554 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:56:24.493063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:24.493031 5367 generic.go:356] "Generic (PLEG): container finished" podID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerID="e67e1bbc9330a8b70f06d7491f94b3d651ed7434bbe32dc382f8bd74249c20af" exitCode=0 Oct 29 22:56:24.493537 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:24.493093 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" event={"ID":"8b18a7d5-0996-425b-96ce-54196e33acbd","Type":"ContainerDied","Data":"e67e1bbc9330a8b70f06d7491f94b3d651ed7434bbe32dc382f8bd74249c20af"} Oct 29 22:56:24.493587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:24.493544 5367 scope.go:122] "RemoveContainer" containerID="e67e1bbc9330a8b70f06d7491f94b3d651ed7434bbe32dc382f8bd74249c20af" Oct 29 22:56:24.495403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:24.495367 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:24.495497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:24.495440 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:24.495497 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:24.495474 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:25.504794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:25.504759 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" event={"ID":"8b18a7d5-0996-425b-96ce-54196e33acbd","Type":"ContainerStarted","Data":"46151889d69c1730d46096ef7b7a59ded5dc61f1e58ceabde89e075efead0f8a"} Oct 29 22:56:26.171723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.160079 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:56:26.179202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.179158 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 29 22:56:26.179378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.179265 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 29 22:56:26.534282 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.534174 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" podUID="aa40ef16-0cce-4852-bc9d-293db5e7dbd0" containerName="payload-pre-processing" containerID="cri-o://8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe" gracePeriod=25 Oct 29 22:56:26.905471 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.905447 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:56:26.989256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.989220 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs\" (UniqueName: \"kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs\") pod \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " Oct 29 22:56:26.989478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.989276 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume\") pod \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\" (UID: \"aa40ef16-0cce-4852-bc9d-293db5e7dbd0\") " Oct 29 22:56:26.989779 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.989750 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume" pod "aa40ef16-0cce-4852-bc9d-293db5e7dbd0" (UID: "aa40ef16-0cce-4852-bc9d-293db5e7dbd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:26.990031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.990012 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:26.992228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:26.992194 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs" pod "aa40ef16-0cce-4852-bc9d-293db5e7dbd0" (UID: "aa40ef16-0cce-4852-bc9d-293db5e7dbd0"). InnerVolumeSpecName "kube-api-access-mcjvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:27.091846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.091809 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjvs\" (UniqueName: \"kubernetes.io/projected/aa40ef16-0cce-4852-bc9d-293db5e7dbd0-kube-api-access-mcjvs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:27.179493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.179385 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 29 22:56:27.179493 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.179455 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 29 22:56:27.523684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.523597 5367 generic.go:356] "Generic (PLEG): container finished" podID="aa40ef16-0cce-4852-bc9d-293db5e7dbd0" containerID="8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe" exitCode=0 Oct 29 22:56:27.523684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.523638 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" Oct 29 22:56:27.523973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.523683 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" event={"ID":"aa40ef16-0cce-4852-bc9d-293db5e7dbd0","Type":"ContainerDied","Data":"8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe"} Oct 29 22:56:27.523973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.523759 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b" event={"ID":"aa40ef16-0cce-4852-bc9d-293db5e7dbd0","Type":"ContainerDied","Data":"066375ee7d89973e241db57874f60b04e32fc460bbae84d94881848a845f807f"} Oct 29 22:56:27.523973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.523789 5367 scope.go:122] "RemoveContainer" containerID="8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe" Oct 29 22:56:27.537273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.537080 5367 scope.go:122] "RemoveContainer" containerID="8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe" Oct 29 22:56:27.537610 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:56:27.537389 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe\": container with ID starting with 8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe not found: ID does not exist" containerID="8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe" Oct 29 22:56:27.537610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.537416 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe"} err="failed to get container status \"8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe\": rpc error: code = NotFound desc = could not find container \"8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe\": container with ID starting with 8a6db8326c41e856cfac642023a9b7eb130feb3aa7a4465bcaebb1baac32babe not found: ID does not exist" Oct 29 22:56:27.603278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.603243 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b"] Oct 29 22:56:27.617095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:27.617065 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-6dcd698c75-csl2b"] Oct 29 22:56:28.173723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:28.167057 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 29 22:56:28.173723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:28.167126 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 29 22:56:28.938143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:28.938109 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:56:29.178791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:29.174114 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 29 22:56:29.178791 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:29.174196 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 29 22:56:29.226642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:29.226560 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa40ef16-0cce-4852-bc9d-293db5e7dbd0" path="/var/lib/kubelet/pods/aa40ef16-0cce-4852-bc9d-293db5e7dbd0/volumes" Oct 29 22:56:30.175185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:30.164103 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 29 22:56:30.175185 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:30.164171 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 29 22:56:31.165022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:31.164987 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:56:31.166128 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:31.166110 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:56:32.034494 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:32.034448 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png"] Oct 29 22:56:32.047191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:32.047157 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff2b4f1-n4png"] Oct 29 22:56:33.224653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:33.224625 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca13b043-9da4-46f0-ac47-338440728a63" path="/var/lib/kubelet/pods/ca13b043-9da4-46f0-ac47-338440728a63/volumes" Oct 29 22:56:39.453103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:39.453057 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v"] Oct 29 22:56:39.464904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:39.464872 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-57d336-r995v"] Oct 29 22:56:40.039304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:40.039266 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj"] Oct 29 22:56:40.052072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:40.052037 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f"] Oct 29 22:56:40.056855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:40.056830 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c23957-vxdlj"] Oct 29 22:56:40.061055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:40.061032 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489b1cc0-njj4f"] Oct 29 22:56:41.266576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:41.266537 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c1b62b-9cb0-4225-bb65-f7bb7eaf179e" path="/var/lib/kubelet/pods/05c1b62b-9cb0-4225-bb65-f7bb7eaf179e/volumes" Oct 29 22:56:41.279642 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:41.279601 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cf1d86-f552-4e77-9fec-9a62e9535ce8" path="/var/lib/kubelet/pods/b4cf1d86-f552-4e77-9fec-9a62e9535ce8/volumes" Oct 29 22:56:41.280576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:41.280549 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce0997e-b6e3-47c6-b1af-d462380ca92d" path="/var/lib/kubelet/pods/dce0997e-b6e3-47c6-b1af-d462380ca92d/volumes" Oct 29 22:56:46.518211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:46.518159 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l"] Oct 29 22:56:46.518780 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:46.518422 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" containerID="cri-o://63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1" gracePeriod=30 Oct 29 22:56:46.755515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:46.755475 5367 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:56:46.755688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:46.755553 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:56:51.809038 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.808971 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:56:51.867159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.867119 5367 generic.go:356] "Generic (PLEG): container finished" podID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerID="63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1" exitCode=0 Oct 29 22:56:51.867344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.867167 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" Oct 29 22:56:51.867344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.867197 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" event={"ID":"0e7d7e47-2202-4ea4-adc7-2804d703c55e","Type":"ContainerDied","Data":"63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1"} Oct 29 22:56:51.867344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.867254 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l" event={"ID":"0e7d7e47-2202-4ea4-adc7-2804d703c55e","Type":"ContainerDied","Data":"6a4a943555f86acced8723b7d96d522449b12a35a0b68976601ad42fcd2e9d35"} Oct 29 22:56:51.867344 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.867276 5367 scope.go:122] "RemoveContainer" containerID="63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1" Oct 29 22:56:51.879210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.879192 5367 scope.go:122] "RemoveContainer" containerID="fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d" Oct 29 22:56:51.899306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.899279 5367 scope.go:122] "RemoveContainer" containerID="63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1" Oct 29 22:56:51.899835 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:56:51.899810 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1\": container with ID starting with 63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1 not found: ID does not exist" containerID="63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1" Oct 29 22:56:51.899955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.899852 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1"} err="failed to get container status \"63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1\": rpc error: code = NotFound desc = could not find container \"63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1\": container with ID starting with 63e37f6305b2284eefc80b337ad5298d6e4d17fa15c9dc80f675618fd9a0d4e1 not found: ID does not exist" Oct 29 22:56:51.899955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.899878 5367 scope.go:122] "RemoveContainer" containerID="fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d" Oct 29 22:56:51.900212 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:56:51.900193 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d\": container with ID starting with fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d not found: ID does not exist" containerID="fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d" Oct 29 22:56:51.900290 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.900222 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d"} err="failed to get container status \"fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d\": rpc error: code = NotFound desc = could not find container \"fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d\": container with ID starting with fed96eaacc838467b6bb0caf9acdce687c1fc2792c0f51ff2706b8b91ca75d3d not found: ID does not exist" Oct 29 22:56:51.907361 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907331 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907380 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907508 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907496 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907538 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907580 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907621 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907716 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907800 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.907966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907883 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:51.907966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907891 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.907966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907911 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.907966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.907949 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data\") pod \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\" (UID: \"0e7d7e47-2202-4ea4-adc7-2804d703c55e\") " Oct 29 22:56:51.909222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.908682 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.909222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.908722 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.909222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.908736 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.909222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.908771 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.909222 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.908945 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.911750 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.911682 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "kube-api-access-j69gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:51.912245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.912222 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:56:51.912548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.912512 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:51.912823 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:51.912804 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy" pod "0e7d7e47-2202-4ea4-adc7-2804d703c55e" (UID: "0e7d7e47-2202-4ea4-adc7-2804d703c55e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:52.010514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.010463 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.010514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.010500 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.010514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.010509 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69gp\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-kube-api-access-j69gp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.010827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.010535 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.010827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.010547 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.010827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.010555 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7d7e47-2202-4ea4-adc7-2804d703c55e-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.034974 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.034940 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh"] Oct 29 22:56:52.039807 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.039782 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c83b040-hj5fh"] Oct 29 22:56:52.219396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.219351 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l"] Oct 29 22:56:52.224399 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:52.224366 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-fgc9l"] Oct 29 22:56:52.334263 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:56:52.334218 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:56:53.225852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:53.225819 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" path="/var/lib/kubelet/pods/0e7d7e47-2202-4ea4-adc7-2804d703c55e/volumes" Oct 29 22:56:53.226525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:53.226503 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36386cd-ba0d-43d2-a610-f0aaed46a1e8" path="/var/lib/kubelet/pods/d36386cd-ba0d-43d2-a610-f0aaed46a1e8/volumes" Oct 29 22:56:57.032243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:57.032201 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6"] Oct 29 22:56:57.044929 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:57.044896 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cefa3b9-4d5s6"] Oct 29 22:56:57.225026 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:56:57.224991 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba88738-149e-4268-a5e9-b354769d568f" path="/var/lib/kubelet/pods/fba88738-149e-4268-a5e9-b354769d568f/volumes" Oct 29 22:57:11.819106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:11.819063 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": read tcp 10.133.0.2:56854->10.133.1.30:15021: read: connection reset by peer" start-of-body= Oct 29 22:57:11.819649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:11.819613 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": read tcp 10.133.0.2:56854->10.133.1.30:15021: read: connection reset by peer" Oct 29 22:57:12.228658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:12.228569 5367 generic.go:356] "Generic (PLEG): container finished" podID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerID="b3b47116b2b94e1c6b7c88fb65b667e4162b88c5bbda5ef9e69839cd5bc34ae4" exitCode=0 Oct 29 22:57:12.228658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:12.228640 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" event={"ID":"c28b49d4-923a-418f-96dc-2cca550cc1e5","Type":"ContainerDied","Data":"b3b47116b2b94e1c6b7c88fb65b667e4162b88c5bbda5ef9e69839cd5bc34ae4"} Oct 29 22:57:12.229110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:12.229092 5367 scope.go:122] "RemoveContainer" containerID="b3b47116b2b94e1c6b7c88fb65b667e4162b88c5bbda5ef9e69839cd5bc34ae4" Oct 29 22:57:12.230907 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:12.230874 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:12.231030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:12.230964 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:12.231030 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:12.231018 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:13.244308 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:13.244269 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" event={"ID":"c28b49d4-923a-418f-96dc-2cca550cc1e5","Type":"ContainerStarted","Data":"bea0bb37ff5ba9be8a31851962ca8d388fcd17075a5a7868a56ef41eaaa72a8c"} Oct 29 22:57:14.197801 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:14.197757 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:57:14.198258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:14.198228 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:57:14.198357 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:14.198302 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:57:15.198398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:15.198361 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.198863 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:15.198420 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:57:16.197819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:16.197784 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:57:16.198018 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:16.197843 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:57:17.198816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:17.198777 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:57:17.199216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:17.198856 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:57:18.198167 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:18.198131 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:57:18.198363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:18.198200 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:57:19.202391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:19.202362 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:57:19.202830 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:19.202812 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:57:19.203504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:19.203483 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 22:57:20.789103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789069 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw"] Oct 29 22:57:20.789817 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789802 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5649e738-d647-4ac7-a618-375f8aa91fc2" containerName="main" Oct 29 22:57:20.789875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789818 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5649e738-d647-4ac7-a618-375f8aa91fc2" containerName="main" Oct 29 22:57:20.789875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789835 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" Oct 29 22:57:20.789875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789841 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" Oct 29 22:57:20.789875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789852 5367 cpu_manager.go:414] "containerMap: removing container" podUID="aa40ef16-0cce-4852-bc9d-293db5e7dbd0" containerName="payload-pre-processing" Oct 29 22:57:20.789875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789858 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa40ef16-0cce-4852-bc9d-293db5e7dbd0" containerName="payload-pre-processing" Oct 29 22:57:20.790041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789884 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" Oct 29 22:57:20.790041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789889 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" Oct 29 22:57:20.790041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.789997 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5649e738-d647-4ac7-a618-375f8aa91fc2" containerName="main" Oct 29 22:57:20.790041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.790007 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" Oct 29 22:57:20.790041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.790015 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa40ef16-0cce-4852-bc9d-293db5e7dbd0" containerName="payload-pre-processing" Oct 29 22:57:20.790041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.790025 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7d7e47-2202-4ea4-adc7-2804d703c55e" containerName="istio-proxy" Oct 29 22:57:20.792900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.792880 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.814105 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.814079 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw"] Oct 29 22:57:20.821680 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.821657 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:20.821850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.821764 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:57:20.887647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.887610 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xdm\" (UniqueName: \"kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.887895 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.887871 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.887965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.887922 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.989510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.989473 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.989752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.989527 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.989752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.989734 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xdm\" (UniqueName: \"kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.990325 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.990297 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:20.992049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:20.992028 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:21.025512 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:21.025486 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xdm\" (UniqueName: \"kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:21.106369 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:21.106339 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:21.480284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:21.479134 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw"] Oct 29 22:57:21.488179 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:57:21.488065 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0d7099_e1c2_47d7_8e3d_bc07de992663.slice/crio-24be1dba40add6f772e9084a261b18d53f8a12a08eb204ef38ba41e34512cc3e WatchSource:0}: Error finding container 24be1dba40add6f772e9084a261b18d53f8a12a08eb204ef38ba41e34512cc3e: Status 404 returned error can't find the container with id 24be1dba40add6f772e9084a261b18d53f8a12a08eb204ef38ba41e34512cc3e Oct 29 22:57:22.334066 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:57:22.334029 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:57:22.437616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:22.437578 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" event={"ID":"1f0d7099-e1c2-47d7-8e3d-bc07de992663","Type":"ContainerStarted","Data":"173737229e1303e675b5608027266272843856a3f491aa7165ff98913b247518"} Oct 29 22:57:22.437840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:22.437635 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" event={"ID":"1f0d7099-e1c2-47d7-8e3d-bc07de992663","Type":"ContainerStarted","Data":"24be1dba40add6f772e9084a261b18d53f8a12a08eb204ef38ba41e34512cc3e"} Oct 29 22:57:22.480550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:22.480496 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" podStartSLOduration=2.480480294 podStartE2EDuration="2.480480294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:22.478877189 +0000 UTC m=+2509.372952124" watchObservedRunningTime="2026-10-29 22:57:22.480480294 +0000 UTC m=+2509.374555216" Oct 29 22:57:23.493725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:23.492331 5367 generic.go:356] "Generic (PLEG): container finished" podID="1f0d7099-e1c2-47d7-8e3d-bc07de992663" containerID="173737229e1303e675b5608027266272843856a3f491aa7165ff98913b247518" exitCode=0 Oct 29 22:57:23.493725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:23.492419 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" event={"ID":"1f0d7099-e1c2-47d7-8e3d-bc07de992663","Type":"ContainerDied","Data":"173737229e1303e675b5608027266272843856a3f491aa7165ff98913b247518"} Oct 29 22:57:24.717937 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.717911 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:24.793716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.793612 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm\" (UniqueName: \"kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm\") pod \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " Oct 29 22:57:24.793716 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.793664 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca\") pod \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " Oct 29 22:57:24.793971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.793767 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca\") pod \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\" (UID: \"1f0d7099-e1c2-47d7-8e3d-bc07de992663\") " Oct 29 22:57:24.794059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.794033 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca" pod "1f0d7099-e1c2-47d7-8e3d-bc07de992663" (UID: "1f0d7099-e1c2-47d7-8e3d-bc07de992663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:24.794514 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.794496 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:24.810532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.810496 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm" pod "1f0d7099-e1c2-47d7-8e3d-bc07de992663" (UID: "1f0d7099-e1c2-47d7-8e3d-bc07de992663"). InnerVolumeSpecName "kube-api-access-b5xdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:24.811905 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.811872 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca" pod "1f0d7099-e1c2-47d7-8e3d-bc07de992663" (UID: "1f0d7099-e1c2-47d7-8e3d-bc07de992663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:24.896371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.896322 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xdm\" (UniqueName: \"kubernetes.io/projected/1f0d7099-e1c2-47d7-8e3d-bc07de992663-kube-api-access-b5xdm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:24.896371 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:24.896373 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0d7099-e1c2-47d7-8e3d-bc07de992663-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:25.485313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:25.485276 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw"] Oct 29 22:57:25.510139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:25.510055 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" Oct 29 22:57:25.510139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:25.510096 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw" event={"ID":"1f0d7099-e1c2-47d7-8e3d-bc07de992663","Type":"ContainerDied","Data":"24be1dba40add6f772e9084a261b18d53f8a12a08eb204ef38ba41e34512cc3e"} Oct 29 22:57:25.510378 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:25.510145 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24be1dba40add6f772e9084a261b18d53f8a12a08eb204ef38ba41e34512cc3e" Oct 29 22:57:26.194774 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.194738 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns"] Oct 29 22:57:26.195599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.195582 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1f0d7099-e1c2-47d7-8e3d-bc07de992663" containerName="revoke-keys" Oct 29 22:57:26.195670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.195601 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0d7099-e1c2-47d7-8e3d-bc07de992663" containerName="revoke-keys" Oct 29 22:57:26.195773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.195762 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0d7099-e1c2-47d7-8e3d-bc07de992663" containerName="revoke-keys" Oct 29 22:57:26.199573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.199550 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.221552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.221521 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-bcb810-openshift-default-dockercfg-78djm\"" Oct 29 22:57:26.223354 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223326 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.223560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223366 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.223560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223474 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.223560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223515 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.223837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223591 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.223837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223669 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.223837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223805 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.224008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223913 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.224008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.223992 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s7cd\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.242025 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.241990 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns"] Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391338 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391445 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391556 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391655 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391749 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s7cd\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391800 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391827 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391935 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.391973 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.392467 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.392490 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.392761 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.392791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.407782 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.393403 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.492783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.487136 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.507625 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.507557 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s7cd\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.560481 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.554038 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.579156 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.579108 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token\") pod \"e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:26.826377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:26.826345 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:27.229206 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:57:27.229169 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933caef7_daeb_4332_97a6_5ea2aedfe708.slice/crio-c84ae7a2063e0ca8393a26685c69a8462cf7f08956ef19735c7546b3b2774f5c WatchSource:0}: Error finding container c84ae7a2063e0ca8393a26685c69a8462cf7f08956ef19735c7546b3b2774f5c: Status 404 returned error can't find the container with id c84ae7a2063e0ca8393a26685c69a8462cf7f08956ef19735c7546b3b2774f5c Oct 29 22:57:27.231210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.231185 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns"] Oct 29 22:57:27.232693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.232661 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:27.232812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.232762 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:27.232812 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.232794 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:27.616684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.616629 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" event={"ID":"933caef7-daeb-4332-97a6-5ea2aedfe708","Type":"ContainerStarted","Data":"a8843d17970f09b33187e5ad49adef68078296495d75b14496fd54494052073f"} Oct 29 22:57:27.616875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.616728 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" event={"ID":"933caef7-daeb-4332-97a6-5ea2aedfe708","Type":"ContainerStarted","Data":"c84ae7a2063e0ca8393a26685c69a8462cf7f08956ef19735c7546b3b2774f5c"} Oct 29 22:57:27.682724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.681186 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podStartSLOduration=1.6811626400000002 podStartE2EDuration="1.68116264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:27.665238607 +0000 UTC m=+2514.559313553" watchObservedRunningTime="2026-10-29 22:57:27.68116264 +0000 UTC m=+2514.575237562" Oct 29 22:57:27.840966 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:27.840180 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:28.834356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:28.834321 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:29.640473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:29.640442 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:29.641736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:29.641679 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:57:30.488720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.487762 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" podUID="0b6c79f8-17d2-4c45-8178-77cb22ab45c6" containerName="payload-pre-processing" containerID="cri-o://0e29a4b17ca838fc86354d63acae1b47e07ab4aaed918a1217630a511c72d7dc" gracePeriod=25 Oct 29 22:57:30.654336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.654290 5367 generic.go:356] "Generic (PLEG): container finished" podID="0b6c79f8-17d2-4c45-8178-77cb22ab45c6" containerID="0e29a4b17ca838fc86354d63acae1b47e07ab4aaed918a1217630a511c72d7dc" exitCode=0 Oct 29 22:57:30.654336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.654321 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" event={"ID":"0b6c79f8-17d2-4c45-8178-77cb22ab45c6","Type":"ContainerDied","Data":"0e29a4b17ca838fc86354d63acae1b47e07ab4aaed918a1217630a511c72d7dc"} Oct 29 22:57:30.829919 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.829246 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:57:30.887112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.887077 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p\" (UniqueName: \"kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p\") pod \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " Oct 29 22:57:30.887310 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.887152 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume\") pod \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\" (UID: \"0b6c79f8-17d2-4c45-8178-77cb22ab45c6\") " Oct 29 22:57:30.888479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.888447 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume" pod "0b6c79f8-17d2-4c45-8178-77cb22ab45c6" (UID: "0b6c79f8-17d2-4c45-8178-77cb22ab45c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:30.893209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.890941 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p" pod "0b6c79f8-17d2-4c45-8178-77cb22ab45c6" (UID: "0b6c79f8-17d2-4c45-8178-77cb22ab45c6"). InnerVolumeSpecName "kube-api-access-m8b9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:30.989612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.989575 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8b9p\" (UniqueName: \"kubernetes.io/projected/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-kube-api-access-m8b9p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:30.989612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:30.989611 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6c79f8-17d2-4c45-8178-77cb22ab45c6-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:31.665165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:31.665137 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" Oct 29 22:57:31.665165 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:31.665160 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw" event={"ID":"0b6c79f8-17d2-4c45-8178-77cb22ab45c6","Type":"ContainerDied","Data":"8ff4d3081e7b6319c50c88eafed9da8d186696c9b7bb05a4a1fc9a57f7e8555a"} Oct 29 22:57:31.665732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:31.665221 5367 scope.go:122] "RemoveContainer" containerID="0e29a4b17ca838fc86354d63acae1b47e07ab4aaed918a1217630a511c72d7dc" Oct 29 22:57:31.733872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:31.733823 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw"] Oct 29 22:57:31.740896 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:31.740864 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-958f44595-dtglw"] Oct 29 22:57:33.227448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.227411 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6c79f8-17d2-4c45-8178-77cb22ab45c6" path="/var/lib/kubelet/pods/0b6c79f8-17d2-4c45-8178-77cb22ab45c6/volumes" Oct 29 22:57:33.738271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.738236 5367 generic.go:356] "Generic (PLEG): container finished" podID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerID="fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c" exitCode=0 Oct 29 22:57:33.738458 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.738288 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" event={"ID":"02720c0f-0336-4df3-8f0c-19853dcd6c59","Type":"ContainerDied","Data":"fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c"} Oct 29 22:57:33.738883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.738862 5367 scope.go:122] "RemoveContainer" containerID="fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c" Oct 29 22:57:33.741000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.740956 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:33.741147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.741041 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:33.741147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:33.741084 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:34.750616 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:34.750581 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" event={"ID":"02720c0f-0336-4df3-8f0c-19853dcd6c59","Type":"ContainerStarted","Data":"98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f"} Oct 29 22:57:35.435915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:35.435879 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:57:35.437166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:35.437136 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.437307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:35.437202 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Oct 29 22:57:35.989724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:35.973877 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:57:36.436090 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:36.436049 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.436316 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:36.436126 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Oct 29 22:57:36.635724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:36.635668 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9"] Oct 29 22:57:36.687525 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:36.687429 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v"] Oct 29 22:57:37.436486 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.436453 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.437005 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.436517 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Oct 29 22:57:37.692487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.692408 5367 scope.go:122] "RemoveContainer" containerID="396a97a1888a9bd1156cd47dc9386a52c83e717a6857301b3843dca16fa13d39" Oct 29 22:57:37.708194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.708171 5367 scope.go:122] "RemoveContainer" containerID="139f97ef551601c0614489897e793aab6a9bbc253605d749bccc800c08505acb" Oct 29 22:57:37.720438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.720415 5367 scope.go:122] "RemoveContainer" containerID="4d3053f6fa4de441e7f4a98ddaff442a5fd8e3e9914de87b3494eab7efcd5054" Oct 29 22:57:37.818792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.808875 5367 scope.go:122] "RemoveContainer" containerID="18b3a278026f232fe85f63987771ea1e3da078018e0f18df7a559a70b552198c" Oct 29 22:57:37.927159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:37.927133 5367 scope.go:122] "RemoveContainer" containerID="8682b2e6034a937531173eb61454b2bf77634abd297f021f79a804b02efdb48b" Oct 29 22:57:38.044555 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:38.044530 5367 scope.go:122] "RemoveContainer" containerID="b9d31bf5c9dc718095c046cdba639afdfdc26c96bea50184320838b89ba51a40" Oct 29 22:57:38.126463 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:38.126435 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw"] Oct 29 22:57:38.141184 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:38.141158 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7a5fe-th9cw"] Oct 29 22:57:38.436095 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:38.436061 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.436284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:38.436116 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Oct 29 22:57:39.226413 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:39.226379 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0d7099-e1c2-47d7-8e3d-bc07de992663" path="/var/lib/kubelet/pods/1f0d7099-e1c2-47d7-8e3d-bc07de992663/volumes" Oct 29 22:57:39.435942 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:39.435897 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Oct 29 22:57:39.436170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:39.435969 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Oct 29 22:57:40.436161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:40.436118 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.436577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:40.436193 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Oct 29 22:57:41.441592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.441554 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:57:41.442731 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.442673 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 22:57:41.636728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.636670 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" podUID="04930aae-6fe9-4061-9676-470076591f5e" containerName="payload-processing" containerID="cri-o://3edf5c46d10f77602ba3c819eddaf712380baafe105e0f021f228b2b9aa3b9c8" gracePeriod=25 Oct 29 22:57:41.687811 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.687751 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" podUID="a80da96b-fb5c-4231-9d06-f65a095081be" containerName="payload-pre-processing" containerID="cri-o://846a276635c8b11bde523c20f9353ce9a9570c8eb4cade445f16d5fa5cf40c8b" gracePeriod=25 Oct 29 22:57:41.981020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.980913 5367 generic.go:356] "Generic (PLEG): container finished" podID="04930aae-6fe9-4061-9676-470076591f5e" containerID="3edf5c46d10f77602ba3c819eddaf712380baafe105e0f021f228b2b9aa3b9c8" exitCode=0 Oct 29 22:57:41.981020 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.980992 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" event={"ID":"04930aae-6fe9-4061-9676-470076591f5e","Type":"ContainerDied","Data":"3edf5c46d10f77602ba3c819eddaf712380baafe105e0f021f228b2b9aa3b9c8"} Oct 29 22:57:41.983747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.983672 5367 generic.go:356] "Generic (PLEG): container finished" podID="a80da96b-fb5c-4231-9d06-f65a095081be" containerID="846a276635c8b11bde523c20f9353ce9a9570c8eb4cade445f16d5fa5cf40c8b" exitCode=0 Oct 29 22:57:41.985001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:41.984940 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" event={"ID":"a80da96b-fb5c-4231-9d06-f65a095081be","Type":"ContainerDied","Data":"846a276635c8b11bde523c20f9353ce9a9570c8eb4cade445f16d5fa5cf40c8b"} Oct 29 22:57:42.014211 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.014179 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:57:42.035577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.035546 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l\" (UniqueName: \"kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l\") pod \"04930aae-6fe9-4061-9676-470076591f5e\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " Oct 29 22:57:42.035795 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.035775 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume\") pod \"04930aae-6fe9-4061-9676-470076591f5e\" (UID: \"04930aae-6fe9-4061-9676-470076591f5e\") " Oct 29 22:57:42.036159 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.036131 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume" pod "04930aae-6fe9-4061-9676-470076591f5e" (UID: "04930aae-6fe9-4061-9676-470076591f5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:42.036760 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.036720 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04930aae-6fe9-4061-9676-470076591f5e-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:42.037865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.037842 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l" pod "04930aae-6fe9-4061-9676-470076591f5e" (UID: "04930aae-6fe9-4061-9676-470076591f5e"). InnerVolumeSpecName "kube-api-access-ljn2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:42.055707 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.055674 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:57:42.137900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.137869 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh\" (UniqueName: \"kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh\") pod \"a80da96b-fb5c-4231-9d06-f65a095081be\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " Oct 29 22:57:42.138193 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.137931 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume\") pod \"a80da96b-fb5c-4231-9d06-f65a095081be\" (UID: \"a80da96b-fb5c-4231-9d06-f65a095081be\") " Oct 29 22:57:42.138462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.138431 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume" pod "a80da96b-fb5c-4231-9d06-f65a095081be" (UID: "a80da96b-fb5c-4231-9d06-f65a095081be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:42.138799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.138777 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn2l\" (UniqueName: \"kubernetes.io/projected/04930aae-6fe9-4061-9676-470076591f5e-kube-api-access-ljn2l\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:42.138799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.138798 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80da96b-fb5c-4231-9d06-f65a095081be-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:42.140243 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.140215 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh" pod "a80da96b-fb5c-4231-9d06-f65a095081be" (UID: "a80da96b-fb5c-4231-9d06-f65a095081be"). InnerVolumeSpecName "kube-api-access-fjbxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:42.240051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.239965 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbxh\" (UniqueName: \"kubernetes.io/projected/a80da96b-fb5c-4231-9d06-f65a095081be-kube-api-access-fjbxh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:42.994772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.994681 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" Oct 29 22:57:42.994772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.994729 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v" event={"ID":"a80da96b-fb5c-4231-9d06-f65a095081be","Type":"ContainerDied","Data":"b264a4c013cd8a89cfecc0dec2eb5513fd555fd99044cd320aeb5f2bac21f9f1"} Oct 29 22:57:42.995314 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.994802 5367 scope.go:122] "RemoveContainer" containerID="846a276635c8b11bde523c20f9353ce9a9570c8eb4cade445f16d5fa5cf40c8b" Oct 29 22:57:42.996649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.996615 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" event={"ID":"04930aae-6fe9-4061-9676-470076591f5e","Type":"ContainerDied","Data":"c7b62cf37536843120dfb68b7f1310e886ea3dc2bfccd41edc94c53562ab90da"} Oct 29 22:57:42.996865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:42.996625 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9" Oct 29 22:57:43.007175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.007152 5367 scope.go:122] "RemoveContainer" containerID="3edf5c46d10f77602ba3c819eddaf712380baafe105e0f021f228b2b9aa3b9c8" Oct 29 22:57:43.051912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.051878 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9"] Oct 29 22:57:43.065592 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.065563 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-586496bcdd-hhlk9"] Oct 29 22:57:43.100448 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.100414 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v"] Oct 29 22:57:43.106306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.106278 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-f97d4446c-hpr6v"] Oct 29 22:57:43.227756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.227235 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04930aae-6fe9-4061-9676-470076591f5e" path="/var/lib/kubelet/pods/04930aae-6fe9-4061-9676-470076591f5e/volumes" Oct 29 22:57:43.228725 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:43.228317 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80da96b-fb5c-4231-9d06-f65a095081be" path="/var/lib/kubelet/pods/a80da96b-fb5c-4231-9d06-f65a095081be/volumes" Oct 29 22:57:47.018349 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.018256 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9"] Oct 29 22:57:47.019231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019207 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0b6c79f8-17d2-4c45-8178-77cb22ab45c6" containerName="payload-pre-processing" Oct 29 22:57:47.019362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019230 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6c79f8-17d2-4c45-8178-77cb22ab45c6" containerName="payload-pre-processing" Oct 29 22:57:47.019362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019254 5367 cpu_manager.go:414] "containerMap: removing container" podUID="04930aae-6fe9-4061-9676-470076591f5e" containerName="payload-processing" Oct 29 22:57:47.019362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019262 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04930aae-6fe9-4061-9676-470076591f5e" containerName="payload-processing" Oct 29 22:57:47.019362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019322 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a80da96b-fb5c-4231-9d06-f65a095081be" containerName="payload-pre-processing" Oct 29 22:57:47.019362 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019330 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80da96b-fb5c-4231-9d06-f65a095081be" containerName="payload-pre-processing" Oct 29 22:57:47.019615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019530 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80da96b-fb5c-4231-9d06-f65a095081be" containerName="payload-pre-processing" Oct 29 22:57:47.019615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019551 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6c79f8-17d2-4c45-8178-77cb22ab45c6" containerName="payload-pre-processing" Oct 29 22:57:47.019615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.019565 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="04930aae-6fe9-4061-9676-470076591f5e" containerName="payload-processing" Oct 29 22:57:47.023391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.023369 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.045461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.045427 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:47.058660 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.058624 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:57:47.068032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.067990 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9"] Oct 29 22:57:47.137247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.137198 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvtg\" (UniqueName: \"kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.137440 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.137273 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.137487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.137461 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.238845 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.238802 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvtg\" (UniqueName: \"kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.239055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.238872 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.239055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.239025 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.239644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.239622 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.241279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.241253 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.263967 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.263934 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvtg\" (UniqueName: \"kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg\") pod \"maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.338560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.338519 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:47.661029 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:47.660998 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9"] Oct 29 22:57:47.661523 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:57:47.661496 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad39e88_8913_4303_8de5_3969aede4a17.slice/crio-0af302c67d43e9d786abc6a5b49df7c7c01796c776f8609444991f4f5e38003f WatchSource:0}: Error finding container 0af302c67d43e9d786abc6a5b49df7c7c01796c776f8609444991f4f5e38003f: Status 404 returned error can't find the container with id 0af302c67d43e9d786abc6a5b49df7c7c01796c776f8609444991f4f5e38003f Oct 29 22:57:48.046066 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:48.045971 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" event={"ID":"fad39e88-8913-4303-8de5-3969aede4a17","Type":"ContainerStarted","Data":"0af302c67d43e9d786abc6a5b49df7c7c01796c776f8609444991f4f5e38003f"} Oct 29 22:57:49.056401 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:49.056358 5367 generic.go:356] "Generic (PLEG): container finished" podID="fad39e88-8913-4303-8de5-3969aede4a17" containerID="62e7e2d56b320855a2c7c01a0e8db2d0357c5ea6db1c9ad80a1fec3769556bba" exitCode=0 Oct 29 22:57:49.056854 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:49.056448 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" event={"ID":"fad39e88-8913-4303-8de5-3969aede4a17","Type":"ContainerDied","Data":"62e7e2d56b320855a2c7c01a0e8db2d0357c5ea6db1c9ad80a1fec3769556bba"} Oct 29 22:57:50.912975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:50.912940 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:51.098111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.098066 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca\") pod \"fad39e88-8913-4303-8de5-3969aede4a17\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " Oct 29 22:57:51.098337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.098130 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca\") pod \"fad39e88-8913-4303-8de5-3969aede4a17\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " Oct 29 22:57:51.098337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.098236 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg\" (UniqueName: \"kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg\") pod \"fad39e88-8913-4303-8de5-3969aede4a17\" (UID: \"fad39e88-8913-4303-8de5-3969aede4a17\") " Oct 29 22:57:51.098523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.098498 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca" pod "fad39e88-8913-4303-8de5-3969aede4a17" (UID: "fad39e88-8913-4303-8de5-3969aede4a17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:51.099408 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.099384 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.114761 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.103484 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg" pod "fad39e88-8913-4303-8de5-3969aede4a17" (UID: "fad39e88-8913-4303-8de5-3969aede4a17"). InnerVolumeSpecName "kube-api-access-kvvtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:51.120799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.120761 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca" pod "fad39e88-8913-4303-8de5-3969aede4a17" (UID: "fad39e88-8913-4303-8de5-3969aede4a17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:51.138118 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.138092 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" Oct 29 22:57:51.138257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.138098 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9" event={"ID":"fad39e88-8913-4303-8de5-3969aede4a17","Type":"ContainerDied","Data":"0af302c67d43e9d786abc6a5b49df7c7c01796c776f8609444991f4f5e38003f"} Oct 29 22:57:51.138257 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.138217 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af302c67d43e9d786abc6a5b49df7c7c01796c776f8609444991f4f5e38003f" Oct 29 22:57:51.201803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.201770 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad39e88-8913-4303-8de5-3969aede4a17-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.201803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:51.201800 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvtg\" (UniqueName: \"kubernetes.io/projected/fad39e88-8913-4303-8de5-3969aede4a17-kube-api-access-kvvtg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:57:52.396554 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:57:52.396509 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:57:57.575931 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.575899 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk"] Oct 29 22:57:57.576628 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.576614 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fad39e88-8913-4303-8de5-3969aede4a17" containerName="revoke-keys" Oct 29 22:57:57.576671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.576629 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad39e88-8913-4303-8de5-3969aede4a17" containerName="revoke-keys" Oct 29 22:57:57.576763 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.576752 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad39e88-8913-4303-8de5-3969aede4a17" containerName="revoke-keys" Oct 29 22:57:57.579653 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.579638 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.608599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.608565 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk"] Oct 29 22:57:57.612622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.612589 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:57.612622 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.612613 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:57:57.626932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.626897 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnbp\" (UniqueName: \"kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.627097 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.626954 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.627238 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.627207 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.728618 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.728576 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnbp\" (UniqueName: \"kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.728814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.728648 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.728814 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.728785 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.729422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.729402 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.731166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.731140 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.750914 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.750888 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnbp\" (UniqueName: \"kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp\") pod \"maas-api-revoke-sub-b615a1bc-rrgzk\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:57.894004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:57.893919 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:57:58.207666 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:57:58.207636 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24914c40_9563_4ca3_9332_cf50abf3a3b0.slice/crio-76570a00fe727941157b2d7a6f1f2439b71ca6c7ade0a66891900444c7670d00 WatchSource:0}: Error finding container 76570a00fe727941157b2d7a6f1f2439b71ca6c7ade0a66891900444c7670d00: Status 404 returned error can't find the container with id 76570a00fe727941157b2d7a6f1f2439b71ca6c7ade0a66891900444c7670d00 Oct 29 22:57:58.208766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:58.208740 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk"] Oct 29 22:57:58.231670 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:58.231632 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" event={"ID":"24914c40-9563-4ca3-9332-cf50abf3a3b0","Type":"ContainerStarted","Data":"76570a00fe727941157b2d7a6f1f2439b71ca6c7ade0a66891900444c7670d00"} Oct 29 22:57:59.243889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:59.243847 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" event={"ID":"24914c40-9563-4ca3-9332-cf50abf3a3b0","Type":"ContainerStarted","Data":"30c4e4b4a8579dfd59724d8b0de4b45d192ef9a13994d3d337a1a761e28b6939"} Oct 29 22:57:59.296115 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:57:59.296047 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" podStartSLOduration=2.296027027 podStartE2EDuration="2.296027027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:59.290806845 +0000 UTC m=+2546.184881766" watchObservedRunningTime="2026-10-29 22:57:59.296027027 +0000 UTC m=+2546.190101952" Oct 29 22:58:00.255102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:00.255066 5367 generic.go:356] "Generic (PLEG): container finished" podID="24914c40-9563-4ca3-9332-cf50abf3a3b0" containerID="30c4e4b4a8579dfd59724d8b0de4b45d192ef9a13994d3d337a1a761e28b6939" exitCode=0 Oct 29 22:58:00.255611 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:00.255156 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" event={"ID":"24914c40-9563-4ca3-9332-cf50abf3a3b0","Type":"ContainerDied","Data":"30c4e4b4a8579dfd59724d8b0de4b45d192ef9a13994d3d337a1a761e28b6939"} Oct 29 22:58:01.265634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.265539 5367 generic.go:356] "Generic (PLEG): container finished" podID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerID="a6ffe6d5198cba2c077811b86643645434dad68d4c6c649de56bbfccaffe1096" exitCode=0 Oct 29 22:58:01.265634 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.265614 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" event={"ID":"0229cbbd-cd97-4074-9d0c-b07aaf7063ba","Type":"ContainerDied","Data":"a6ffe6d5198cba2c077811b86643645434dad68d4c6c649de56bbfccaffe1096"} Oct 29 22:58:01.266284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.266109 5367 scope.go:122] "RemoveContainer" containerID="a6ffe6d5198cba2c077811b86643645434dad68d4c6c649de56bbfccaffe1096" Oct 29 22:58:01.268032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.268003 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:01.268114 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.268072 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:01.268152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.268116 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:01.463209 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.463183 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:58:01.609092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.609054 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp\" (UniqueName: \"kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp\") pod \"24914c40-9563-4ca3-9332-cf50abf3a3b0\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " Oct 29 22:58:01.609317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.609258 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca\") pod \"24914c40-9563-4ca3-9332-cf50abf3a3b0\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " Oct 29 22:58:01.609483 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.609459 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca\") pod \"24914c40-9563-4ca3-9332-cf50abf3a3b0\" (UID: \"24914c40-9563-4ca3-9332-cf50abf3a3b0\") " Oct 29 22:58:01.609865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.609839 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca" pod "24914c40-9563-4ca3-9332-cf50abf3a3b0" (UID: "24914c40-9563-4ca3-9332-cf50abf3a3b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:01.610202 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.610175 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:01.611455 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.611428 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca" pod "24914c40-9563-4ca3-9332-cf50abf3a3b0" (UID: "24914c40-9563-4ca3-9332-cf50abf3a3b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:01.611557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.611496 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp" pod "24914c40-9563-4ca3-9332-cf50abf3a3b0" (UID: "24914c40-9563-4ca3-9332-cf50abf3a3b0"). InnerVolumeSpecName "kube-api-access-htnbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:01.712315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.712273 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnbp\" (UniqueName: \"kubernetes.io/projected/24914c40-9563-4ca3-9332-cf50abf3a3b0-kube-api-access-htnbp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:01.712529 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:01.712321 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24914c40-9563-4ca3-9332-cf50abf3a3b0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:02.275749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.275715 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" Oct 29 22:58:02.275749 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.275727 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk" event={"ID":"24914c40-9563-4ca3-9332-cf50abf3a3b0","Type":"ContainerDied","Data":"76570a00fe727941157b2d7a6f1f2439b71ca6c7ade0a66891900444c7670d00"} Oct 29 22:58:02.276235 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.275781 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76570a00fe727941157b2d7a6f1f2439b71ca6c7ade0a66891900444c7670d00" Oct 29 22:58:02.278036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.278004 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" event={"ID":"0229cbbd-cd97-4074-9d0c-b07aaf7063ba","Type":"ContainerStarted","Data":"868f97a08ff31a632a0d4dc8ebf32d10bbc64a96c7be9db4281147f5fba1c1d9"} Oct 29 22:58:02.438923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.438892 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk"] Oct 29 22:58:02.439152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.439075 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" containerID="cri-o://3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21" gracePeriod=30 Oct 29 22:58:02.817799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.817756 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:58:02.819407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.819379 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 29 22:58:02.819565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:02.819444 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 29 22:58:03.132245 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:03.132162 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:58:03.818169 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:03.818134 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 29 22:58:03.818602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:03.818188 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 29 22:58:04.818583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:04.818548 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 29 22:58:04.819059 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:04.818614 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 29 22:58:05.818241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:05.818203 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 29 22:58:05.818461 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:05.818274 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 29 22:58:06.184223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:06.184138 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:58:06.184223 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:06.184205 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:58:06.817987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:06.817950 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 29 22:58:06.818179 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:06.818001 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 29 22:58:07.819856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.818533 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 29 22:58:07.819856 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.818646 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 29 22:58:07.886689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.886657 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:58:07.898613 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.898584 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z"] Oct 29 22:58:07.899366 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.899350 5367 cpu_manager.go:414] "containerMap: removing container" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" Oct 29 22:58:07.899422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.899368 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" Oct 29 22:58:07.899422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.899392 5367 cpu_manager.go:414] "containerMap: removing container" podUID="24914c40-9563-4ca3-9332-cf50abf3a3b0" containerName="revoke-keys" Oct 29 22:58:07.899422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.899397 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24914c40-9563-4ca3-9332-cf50abf3a3b0" containerName="revoke-keys" Oct 29 22:58:07.899531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.899500 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerName="istio-proxy" Oct 29 22:58:07.899531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.899512 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="24914c40-9563-4ca3-9332-cf50abf3a3b0" containerName="revoke-keys" Oct 29 22:58:07.902143 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.902127 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:07.919069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.919045 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:58:07.919069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.919059 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:58:07.925281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.925252 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z"] Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951436 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951508 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951623 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951668 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951779 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951837 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951896 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.951938 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.952186 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket\") pod \"950c8f41-a5b6-483a-a31b-2edc57387dfe\" (UID: \"950c8f41-a5b6-483a-a31b-2edc57387dfe\") " Oct 29 22:58:07.952917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.952591 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:07.952917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.952878 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:07.953032 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.952943 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrwn\" (UniqueName: \"kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:07.953467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.953438 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:07.953804 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.953776 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:07.968857 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.968816 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:07.972816 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.969581 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:07.978220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.970798 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:07.978518 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.975917 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:07.982112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.982079 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:58:07.982627 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.982604 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "kube-api-access-zqcgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:07.982984 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:07.982961 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy" pod "950c8f41-a5b6-483a-a31b-2edc57387dfe" (UID: "950c8f41-a5b6-483a-a31b-2edc57387dfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:08.058313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.055360 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.058313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.055977 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrwn\" (UniqueName: \"kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.058313 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.056180 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059488 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059517 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcgc\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-kube-api-access-zqcgc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059532 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059545 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059557 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059569 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059581 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c8f41-a5b6-483a-a31b-2edc57387dfe-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059595 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c8f41-a5b6-483a-a31b-2edc57387dfe-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.059609 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c8f41-a5b6-483a-a31b-2edc57387dfe-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:08.061234 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.060409 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.064058 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.064035 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.083441 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.083416 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrwn\" (UniqueName: \"kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn\") pod \"maas-api-revoke-sub-bb98897d-8x25z\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.215145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.215112 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:08.337837 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.337805 5367 generic.go:356] "Generic (PLEG): container finished" podID="950c8f41-a5b6-483a-a31b-2edc57387dfe" containerID="3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21" exitCode=0 Oct 29 22:58:08.338024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.337844 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" event={"ID":"950c8f41-a5b6-483a-a31b-2edc57387dfe","Type":"ContainerDied","Data":"3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21"} Oct 29 22:58:08.338024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.337892 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" Oct 29 22:58:08.338024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.337905 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk" event={"ID":"950c8f41-a5b6-483a-a31b-2edc57387dfe","Type":"ContainerDied","Data":"6aa01bb1595f52828e656a18726334c62d0af452f683bc5e359f4398008ffbf6"} Oct 29 22:58:08.338024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.337931 5367 scope.go:122] "RemoveContainer" containerID="3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21" Oct 29 22:58:08.350227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.350203 5367 scope.go:122] "RemoveContainer" containerID="3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21" Oct 29 22:58:08.350735 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:58:08.350609 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21\": container with ID starting with 3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21 not found: ID does not exist" containerID="3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21" Oct 29 22:58:08.350735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.350641 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21"} err="failed to get container status \"3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21\": rpc error: code = NotFound desc = could not find container \"3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21\": container with ID starting with 3f6ace67696023ce8de955ffa881311e2fda8ae8f0c22f7c124ccd49c225ac21 not found: ID does not exist" Oct 29 22:58:08.402216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.402180 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk"] Oct 29 22:58:08.407099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.407072 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zwdk"] Oct 29 22:58:08.522864 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:08.522825 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7aab1d_ff34_4119_a84f_68fb519029d2.slice/crio-02dd17734bf2134afbf8fa3b0fbaf6e522c5f71908e71e7d017acb9263412d2f WatchSource:0}: Error finding container 02dd17734bf2134afbf8fa3b0fbaf6e522c5f71908e71e7d017acb9263412d2f: Status 404 returned error can't find the container with id 02dd17734bf2134afbf8fa3b0fbaf6e522c5f71908e71e7d017acb9263412d2f Oct 29 22:58:08.523492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.523468 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z"] Oct 29 22:58:08.823293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.823260 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:58:08.824528 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:08.824504 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 22:58:09.228328 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:09.227961 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950c8f41-a5b6-483a-a31b-2edc57387dfe" path="/var/lib/kubelet/pods/950c8f41-a5b6-483a-a31b-2edc57387dfe/volumes" Oct 29 22:58:09.420301 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:09.420263 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" event={"ID":"3c7aab1d-ff34-4119-a84f-68fb519029d2","Type":"ContainerStarted","Data":"fd1b438a32f7400b0a131fc078402f09cfb159689024dc1b02081f3185d55273"} Oct 29 22:58:09.420575 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:09.420549 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" event={"ID":"3c7aab1d-ff34-4119-a84f-68fb519029d2","Type":"ContainerStarted","Data":"02dd17734bf2134afbf8fa3b0fbaf6e522c5f71908e71e7d017acb9263412d2f"} Oct 29 22:58:09.474819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:09.474757 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" podStartSLOduration=2.474737897 podStartE2EDuration="2.474737897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:09.468065521 +0000 UTC m=+2556.362140442" watchObservedRunningTime="2026-10-29 22:58:09.474737897 +0000 UTC m=+2556.368812821" Oct 29 22:58:10.424200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:10.424155 5367 generic.go:356] "Generic (PLEG): container finished" podID="3c7aab1d-ff34-4119-a84f-68fb519029d2" containerID="fd1b438a32f7400b0a131fc078402f09cfb159689024dc1b02081f3185d55273" exitCode=0 Oct 29 22:58:10.424692 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:10.424241 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" event={"ID":"3c7aab1d-ff34-4119-a84f-68fb519029d2","Type":"ContainerDied","Data":"fd1b438a32f7400b0a131fc078402f09cfb159689024dc1b02081f3185d55273"} Oct 29 22:58:11.615693 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.615659 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:11.753070 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.752976 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca\") pod \"3c7aab1d-ff34-4119-a84f-68fb519029d2\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " Oct 29 22:58:11.753279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.753181 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca\") pod \"3c7aab1d-ff34-4119-a84f-68fb519029d2\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " Oct 29 22:58:11.753279 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.753207 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn\" (UniqueName: \"kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn\") pod \"3c7aab1d-ff34-4119-a84f-68fb519029d2\" (UID: \"3c7aab1d-ff34-4119-a84f-68fb519029d2\") " Oct 29 22:58:11.753438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.753412 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca" pod "3c7aab1d-ff34-4119-a84f-68fb519029d2" (UID: "3c7aab1d-ff34-4119-a84f-68fb519029d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:11.753747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.753728 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:11.755329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.755301 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca" pod "3c7aab1d-ff34-4119-a84f-68fb519029d2" (UID: "3c7aab1d-ff34-4119-a84f-68fb519029d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:11.755571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.755548 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn" pod "3c7aab1d-ff34-4119-a84f-68fb519029d2" (UID: "3c7aab1d-ff34-4119-a84f-68fb519029d2"). InnerVolumeSpecName "kube-api-access-xkrwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:11.855422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.855382 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7aab1d-ff34-4119-a84f-68fb519029d2-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:11.855422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:11.855414 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrwn\" (UniqueName: \"kubernetes.io/projected/3c7aab1d-ff34-4119-a84f-68fb519029d2-kube-api-access-xkrwn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:12.443504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:12.443479 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" Oct 29 22:58:12.443504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:12.443491 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z" event={"ID":"3c7aab1d-ff34-4119-a84f-68fb519029d2","Type":"ContainerDied","Data":"02dd17734bf2134afbf8fa3b0fbaf6e522c5f71908e71e7d017acb9263412d2f"} Oct 29 22:58:12.443754 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:12.443533 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02dd17734bf2134afbf8fa3b0fbaf6e522c5f71908e71e7d017acb9263412d2f" Oct 29 22:58:13.108256 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.108216 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4"] Oct 29 22:58:13.109022 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.109005 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3c7aab1d-ff34-4119-a84f-68fb519029d2" containerName="revoke-keys" Oct 29 22:58:13.109082 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.109024 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7aab1d-ff34-4119-a84f-68fb519029d2" containerName="revoke-keys" Oct 29 22:58:13.109145 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.109135 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7aab1d-ff34-4119-a84f-68fb519029d2" containerName="revoke-keys" Oct 29 22:58:13.111883 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.111861 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.151726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.140942 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-36b46d-openshift-default-dockercfg-9qfvs\"" Oct 29 22:58:13.194396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.194355 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4"] Oct 29 22:58:13.205355 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205322 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205371 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205526 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205390 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205568 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205689 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205613 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205790 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205856 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.205920 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.205885 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wpzz\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.206048 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.206013 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.307946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.307901 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308007 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308061 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308085 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308201 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308339 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308234 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308336 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308442 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308403 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308434 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wpzz\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308552 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308505 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308610 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.308794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.308757 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.309034 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.309009 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.309085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.309070 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.310585 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.310564 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.310917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.310897 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.333107 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.333079 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.333946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.333925 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wpzz\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.445880 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.445793 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:13.780803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.780753 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4"] Oct 29 22:58:13.783098 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:13.782614 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be767f7_8da0_401b_8e43_caef5ec3fce7.slice/crio-e4feccd54f7a5fe2d0e80a96ad76f0360e783f4b07500df867be75e3e6310d0b WatchSource:0}: Error finding container e4feccd54f7a5fe2d0e80a96ad76f0360e783f4b07500df867be75e3e6310d0b: Status 404 returned error can't find the container with id e4feccd54f7a5fe2d0e80a96ad76f0360e783f4b07500df867be75e3e6310d0b Oct 29 22:58:13.786745 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.786687 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:13.786864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.786794 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:13.786864 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:13.786847 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:14.462935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.462896 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" event={"ID":"5be767f7-8da0-401b-8e43-caef5ec3fce7","Type":"ContainerStarted","Data":"b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f"} Oct 29 22:58:14.463397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.462956 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" event={"ID":"5be767f7-8da0-401b-8e43-caef5ec3fce7","Type":"ContainerStarted","Data":"e4feccd54f7a5fe2d0e80a96ad76f0360e783f4b07500df867be75e3e6310d0b"} Oct 29 22:58:14.511851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.511795 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podStartSLOduration=1.511780242 podStartE2EDuration="1.511780242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:14.51091148 +0000 UTC m=+2561.404986402" watchObservedRunningTime="2026-10-29 22:58:14.511780242 +0000 UTC m=+2561.405855165" Oct 29 22:58:14.603947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.603907 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj"] Oct 29 22:58:14.607571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.607549 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.628247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.627865 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-329f7b-openshift-default-dockercfg-fvqhf\"" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630205 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630259 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630304 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630445 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630495 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630554 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.630583 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.631109 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2nw\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.631722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.631178 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.641776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.634988 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj"] Oct 29 22:58:14.732898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.732813 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.732898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.732857 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.732913 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.732937 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733108 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2nw\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733159 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733198 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733232 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733273 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733549 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733303 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733614 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733550 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733680 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733736 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733690 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.733877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.733861 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.735412 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.735391 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.735637 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.735616 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.782999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.782964 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2nw\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.783965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.783935 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:14.929403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:14.929360 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:15.287680 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:15.287642 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f12f3fb_0d53_4b06_a490_c93a9510aa91.slice/crio-be6ff00f4342e45b55e79d9ec4f046cf146e61467b3ab2be5a42fbe174db9548 WatchSource:0}: Error finding container be6ff00f4342e45b55e79d9ec4f046cf146e61467b3ab2be5a42fbe174db9548: Status 404 returned error can't find the container with id be6ff00f4342e45b55e79d9ec4f046cf146e61467b3ab2be5a42fbe174db9548 Oct 29 22:58:15.290897 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.290853 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:15.291031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.290927 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:15.291031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.290956 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:15.299718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.299679 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj"] Oct 29 22:58:15.446951 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.446905 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:15.448288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.448261 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.448436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.448307 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Oct 29 22:58:15.490122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.490085 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" event={"ID":"2f12f3fb-0d53-4b06-a490-c93a9510aa91","Type":"ContainerStarted","Data":"e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1"} Oct 29 22:58:15.490643 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.490151 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" event={"ID":"2f12f3fb-0d53-4b06-a490-c93a9510aa91","Type":"ContainerStarted","Data":"be6ff00f4342e45b55e79d9ec4f046cf146e61467b3ab2be5a42fbe174db9548"} Oct 29 22:58:15.570785 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.557362 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podStartSLOduration=1.5573395410000002 podStartE2EDuration="1.557339541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:15.554343882 +0000 UTC m=+2562.448418815" watchObservedRunningTime="2026-10-29 22:58:15.557339541 +0000 UTC m=+2562.451414464" Oct 29 22:58:15.930560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:15.930443 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:16.047964 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.047929 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx"] Oct 29 22:58:16.089094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.089046 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac6b1fd-mjcmx"] Oct 29 22:58:16.094111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.094078 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c"] Oct 29 22:58:16.101175 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.101140 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a03bfd5-sth5c"] Oct 29 22:58:16.446353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.446317 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.446554 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.446367 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Oct 29 22:58:16.931781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.931643 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.931781 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:16.931741 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:58:17.229523 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:17.229444 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417c6534-7643-49d6-a44d-de910b7978ba" path="/var/lib/kubelet/pods/417c6534-7643-49d6-a44d-de910b7978ba/volumes" Oct 29 22:58:17.229946 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:17.229930 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f08ea58-e238-47f6-b054-7795c8916a2e" path="/var/lib/kubelet/pods/9f08ea58-e238-47f6-b054-7795c8916a2e/volumes" Oct 29 22:58:17.446623 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:17.446587 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.446861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:17.446654 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Oct 29 22:58:17.930063 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:17.930005 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.930287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:17.930076 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:58:18.447146 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:18.447107 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Oct 29 22:58:18.447580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:18.447165 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Oct 29 22:58:19.003723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:19.003588 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:58:19.003723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:19.003654 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:58:19.446615 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:19.446580 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Oct 29 22:58:19.446818 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:19.446637 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Oct 29 22:58:19.930312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:19.930262 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:58:19.930850 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:19.930338 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:58:20.447227 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:20.447192 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Oct 29 22:58:20.447429 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:20.447252 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Oct 29 22:58:20.930092 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:20.930048 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:58:20.930346 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:20.930109 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:58:21.451939 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:21.451900 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:21.452375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:21.452363 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:21.453214 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:21.453192 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 22:58:21.930299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:21.930254 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:58:21.930492 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:21.930320 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:58:22.339446 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:58:22.339402 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:58:22.979633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:22.979593 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:22.981965 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:22.981944 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:22.986876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:22.986854 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 22:58:26.032950 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:26.032915 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm"] Oct 29 22:58:26.046384 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:26.046352 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms"] Oct 29 22:58:26.051363 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:26.051333 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d0c7143-gxxmm"] Oct 29 22:58:26.055583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:26.055559 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e70fd8e-c68ms"] Oct 29 22:58:27.225013 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:27.224981 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce10230-33d1-4a11-bcc5-4abbcd119011" path="/var/lib/kubelet/pods/9ce10230-33d1-4a11-bcc5-4abbcd119011/volumes" Oct 29 22:58:27.225505 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:27.225490 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6935b0d-814d-4678-a78c-86f548e3465f" path="/var/lib/kubelet/pods/a6935b0d-814d-4678-a78c-86f548e3465f/volumes" Oct 29 22:58:33.209865 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.209824 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2"] Oct 29 22:58:33.210374 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.210079 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" podUID="1d36e7b2-183b-473f-bdb6-3a1f38c63efe" containerName="maas-api" containerID="cri-o://395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3" gracePeriod=30 Oct 29 22:58:33.562996 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.562970 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:58:33.594794 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.594760 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs\") pod \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " Oct 29 22:58:33.594956 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.594916 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\") pod \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " Oct 29 22:58:33.595024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.594966 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb\" (UniqueName: \"kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb\") pod \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " Oct 29 22:58:33.595024 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.594988 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca\") pod \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\" (UID: \"1d36e7b2-183b-473f-bdb6-3a1f38c63efe\") " Oct 29 22:58:33.595426 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.595396 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca" pod "1d36e7b2-183b-473f-bdb6-3a1f38c63efe" (UID: "1d36e7b2-183b-473f-bdb6-3a1f38c63efe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:33.596017 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.595989 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:33.597077 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.597045 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs" pod "1d36e7b2-183b-473f-bdb6-3a1f38c63efe" (UID: "1d36e7b2-183b-473f-bdb6-3a1f38c63efe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:33.597296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.597269 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls" pod "1d36e7b2-183b-473f-bdb6-3a1f38c63efe" (UID: "1d36e7b2-183b-473f-bdb6-3a1f38c63efe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:33.597345 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.597290 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb" pod "1d36e7b2-183b-473f-bdb6-3a1f38c63efe" (UID: "1d36e7b2-183b-473f-bdb6-3a1f38c63efe"). InnerVolumeSpecName "kube-api-access-jbkmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:33.697196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.697157 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:33.697196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.697189 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:33.697196 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.697199 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbkmb\" (UniqueName: \"kubernetes.io/projected/1d36e7b2-183b-473f-bdb6-3a1f38c63efe-kube-api-access-jbkmb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:33.822734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.820858 5367 generic.go:356] "Generic (PLEG): container finished" podID="1d36e7b2-183b-473f-bdb6-3a1f38c63efe" containerID="395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3" exitCode=0 Oct 29 22:58:33.822734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.821052 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" event={"ID":"1d36e7b2-183b-473f-bdb6-3a1f38c63efe","Type":"ContainerDied","Data":"395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3"} Oct 29 22:58:33.822734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.821103 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" event={"ID":"1d36e7b2-183b-473f-bdb6-3a1f38c63efe","Type":"ContainerDied","Data":"e53d478231400abef875f4b6f68d113eed120842bcf7b024911caca722c5a4bb"} Oct 29 22:58:33.822734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.821134 5367 scope.go:122] "RemoveContainer" containerID="395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3" Oct 29 22:58:33.822734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.821295 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2" Oct 29 22:58:33.886924 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.886834 5367 scope.go:122] "RemoveContainer" containerID="395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3" Oct 29 22:58:33.888769 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:58:33.887553 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3\": container with ID starting with 395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3 not found: ID does not exist" containerID="395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3" Oct 29 22:58:33.888769 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.887583 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3"} err="failed to get container status \"395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3\": rpc error: code = NotFound desc = could not find container \"395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3\": container with ID starting with 395360dc8dfef3fb486d9eb794d630a7a5e8abe9ec5cc21f15a062dc9ba1b9f3 not found: ID does not exist" Oct 29 22:58:33.933723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.929400 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2"] Oct 29 22:58:33.966724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:33.956417 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09fc89-59f87c5fd5-xvsc2"] Oct 29 22:58:34.039925 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:34.039890 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9"] Oct 29 22:58:34.044419 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:34.044394 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09fc89-f8ft9"] Oct 29 22:58:35.225385 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:35.225340 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d36e7b2-183b-473f-bdb6-3a1f38c63efe" path="/var/lib/kubelet/pods/1d36e7b2-183b-473f-bdb6-3a1f38c63efe/volumes" Oct 29 22:58:35.226108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:35.226086 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad39e88-8913-4303-8de5-3969aede4a17" path="/var/lib/kubelet/pods/fad39e88-8913-4303-8de5-3969aede4a17/volumes" Oct 29 22:58:38.031648 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.031609 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29"] Oct 29 22:58:38.044510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.044481 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f927f65-clm29"] Oct 29 22:58:38.141656 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.141624 5367 scope.go:122] "RemoveContainer" containerID="113439ea84f9d18703bbe6e4766eb04b308a241b4aac8b542c8658ef0d038ede" Oct 29 22:58:38.153224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.153195 5367 scope.go:122] "RemoveContainer" containerID="7aa5e59a9f007ada6fe83e197b589fbcdbd32e85dca77f3f8194728fe5bbeade" Oct 29 22:58:38.163900 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.163872 5367 scope.go:122] "RemoveContainer" containerID="fd5ec22391c405b22eff8cc47b97f33c3cf155c0f6e79ed524c0e0f7cd79659c" Oct 29 22:58:38.174646 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.174624 5367 scope.go:122] "RemoveContainer" containerID="8608d6372fd976138f997bbf0dc804cb77a85dd4b36dfe38fdbe852bf8f0d828" Oct 29 22:58:38.186499 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.186469 5367 scope.go:122] "RemoveContainer" containerID="92cf6ecd67c1affc316b7b98a019332a214e9b87440c3949d0ac7a970c7d09c3" Oct 29 22:58:38.196912 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:38.196889 5367 scope.go:122] "RemoveContainer" containerID="ad4e5cf196eb65f9653534de5379599092d6f50808e3c66e61c9b578e3516069" Oct 29 22:58:39.233397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:39.233344 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ce5d3d-8626-4f65-9f76-ddfe500d2a66" path="/var/lib/kubelet/pods/76ce5d3d-8626-4f65-9f76-ddfe500d2a66/volumes" Oct 29 22:58:40.338236 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.338205 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp"] Oct 29 22:58:40.338958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.338941 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1d36e7b2-183b-473f-bdb6-3a1f38c63efe" containerName="maas-api" Oct 29 22:58:40.339023 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.338959 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d36e7b2-183b-473f-bdb6-3a1f38c63efe" containerName="maas-api" Oct 29 22:58:40.339102 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.339093 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d36e7b2-183b-473f-bdb6-3a1f38c63efe" containerName="maas-api" Oct 29 22:58:40.344981 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.344963 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.362456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.362435 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0906f4-openshift-default-dockercfg-6mxbq\"" Oct 29 22:58:40.395995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.395969 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp"] Oct 29 22:58:40.427317 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427237 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.427473 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427378 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.427589 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427520 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.427852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427666 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.427852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427754 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghs58\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.427852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427799 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.427852 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427834 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.428154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.427956 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.428154 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.428037 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.643368 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.643473 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.647327 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.647396 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.647534 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.654523 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.654595 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghs58\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.654651 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.654688 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.656962 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.657413 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.659916 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.661773 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.660917 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.663522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.663469 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.682877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.673195 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.683329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.683299 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.697738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.692831 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.697738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.694320 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghs58\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58\") pod \"e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:40.959432 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:40.959349 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:41.284249 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:41.284209 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466bb494_cc30_4ab1_b12d_b6157e64f152.slice/crio-d76ee21bcfb29d02ef0abb594678cb3f22316a71356288d6f96b0a9fc3b02d05 WatchSource:0}: Error finding container d76ee21bcfb29d02ef0abb594678cb3f22316a71356288d6f96b0a9fc3b02d05: Status 404 returned error can't find the container with id d76ee21bcfb29d02ef0abb594678cb3f22316a71356288d6f96b0a9fc3b02d05 Oct 29 22:58:41.284377 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:41.284335 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp"] Oct 29 22:58:41.287577 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:41.287544 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:41.287671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:41.287610 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:41.287671 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:41.287640 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:41.752711 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:41.752662 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp"] Oct 29 22:58:41.753116 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:41.752895 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" containerID="cri-o://46151889d69c1730d46096ef7b7a59ded5dc61f1e58ceabde89e075efead0f8a" gracePeriod=30 Oct 29 22:58:42.051150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.046059 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" event={"ID":"466bb494-cc30-4ab1-b12d-b6157e64f152","Type":"ContainerStarted","Data":"69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8"} Oct 29 22:58:42.051150 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.046130 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" event={"ID":"466bb494-cc30-4ab1-b12d-b6157e64f152","Type":"ContainerStarted","Data":"d76ee21bcfb29d02ef0abb594678cb3f22316a71356288d6f96b0a9fc3b02d05"} Oct 29 22:58:42.101293 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.101245 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podStartSLOduration=2.101230359 podStartE2EDuration="2.101230359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:42.097283348 +0000 UTC m=+2588.991358272" watchObservedRunningTime="2026-10-29 22:58:42.101230359 +0000 UTC m=+2588.995305280" Oct 29 22:58:42.201600 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.201567 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x"] Oct 29 22:58:42.206231 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.206209 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.232472 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.232136 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:58:42.243720 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.243212 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:58:42.289724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.286156 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x"] Oct 29 22:58:42.351210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.351174 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.351210 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.351213 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.351855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.351821 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmdjc\" (UniqueName: \"kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.453074 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.453044 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmdjc\" (UniqueName: \"kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.453241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.453092 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.453241 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.453115 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.453889 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.453862 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.455467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.455446 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.474327 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.474303 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmdjc\" (UniqueName: \"kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.585550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.585520 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:42.899619 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:42.899589 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7782b4_7a51_4e8b_b966_101bae1cb930.slice/crio-0b5dd9fec1994d41ea2c71255c88f5424a081170f5c95d04cdb3484f33ebd287 WatchSource:0}: Error finding container 0b5dd9fec1994d41ea2c71255c88f5424a081170f5c95d04cdb3484f33ebd287: Status 404 returned error can't find the container with id 0b5dd9fec1994d41ea2c71255c88f5424a081170f5c95d04cdb3484f33ebd287 Oct 29 22:58:42.900071 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.899947 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x"] Oct 29 22:58:42.960291 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.960267 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:42.967065 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:42.967040 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:43.062276 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:43.062162 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" event={"ID":"dd7782b4-7a51-4e8b-b966-101bae1cb930","Type":"ContainerStarted","Data":"0b5dd9fec1994d41ea2c71255c88f5424a081170f5c95d04cdb3484f33ebd287"} Oct 29 22:58:43.062558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:43.062514 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:43.063587 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:43.063566 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 22:58:44.073932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:44.073899 5367 generic.go:356] "Generic (PLEG): container finished" podID="dd7782b4-7a51-4e8b-b966-101bae1cb930" containerID="977bbe9bf7ec6527fb97dddaae11869204c0f91322eb07147a8d84c6e698158a" exitCode=0 Oct 29 22:58:44.074329 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:44.073987 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" event={"ID":"dd7782b4-7a51-4e8b-b966-101bae1cb930","Type":"ContainerDied","Data":"977bbe9bf7ec6527fb97dddaae11869204c0f91322eb07147a8d84c6e698158a"} Oct 29 22:58:45.265620 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.265587 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:45.406991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.406913 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca\") pod \"dd7782b4-7a51-4e8b-b966-101bae1cb930\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " Oct 29 22:58:45.406991 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.406960 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc\" (UniqueName: \"kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc\") pod \"dd7782b4-7a51-4e8b-b966-101bae1cb930\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " Oct 29 22:58:45.407397 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.407371 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca\") pod \"dd7782b4-7a51-4e8b-b966-101bae1cb930\" (UID: \"dd7782b4-7a51-4e8b-b966-101bae1cb930\") " Oct 29 22:58:45.407838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.407807 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca" pod "dd7782b4-7a51-4e8b-b966-101bae1cb930" (UID: "dd7782b4-7a51-4e8b-b966-101bae1cb930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:45.408049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.408030 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:45.409168 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.409145 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca" pod "dd7782b4-7a51-4e8b-b966-101bae1cb930" (UID: "dd7782b4-7a51-4e8b-b966-101bae1cb930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:45.409294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.409274 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc" pod "dd7782b4-7a51-4e8b-b966-101bae1cb930" (UID: "dd7782b4-7a51-4e8b-b966-101bae1cb930"). InnerVolumeSpecName "kube-api-access-mmdjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:45.509126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.509094 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7782b4-7a51-4e8b-b966-101bae1cb930-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:45.509126 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:45.509122 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmdjc\" (UniqueName: \"kubernetes.io/projected/dd7782b4-7a51-4e8b-b966-101bae1cb930-kube-api-access-mmdjc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:46.105960 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.105925 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" Oct 29 22:58:46.106136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.105957 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x" event={"ID":"dd7782b4-7a51-4e8b-b966-101bae1cb930","Type":"ContainerDied","Data":"0b5dd9fec1994d41ea2c71255c88f5424a081170f5c95d04cdb3484f33ebd287"} Oct 29 22:58:46.106136 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.106047 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5dd9fec1994d41ea2c71255c88f5424a081170f5c95d04cdb3484f33ebd287" Oct 29 22:58:46.168431 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.168393 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:58:46.168595 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.168456 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:58:46.717224 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.717185 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd"] Oct 29 22:58:46.718132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.718113 5367 cpu_manager.go:414] "containerMap: removing container" podUID="dd7782b4-7a51-4e8b-b966-101bae1cb930" containerName="revoke-keys" Oct 29 22:58:46.718201 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.718134 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7782b4-7a51-4e8b-b966-101bae1cb930" containerName="revoke-keys" Oct 29 22:58:46.718272 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.718263 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7782b4-7a51-4e8b-b966-101bae1cb930" containerName="revoke-keys" Oct 29 22:58:46.721726 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.721687 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:46.743772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.743747 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0906f4-dockercfg-xtcfn\"" Oct 29 22:58:46.750072 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.750046 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd"] Oct 29 22:58:46.755980 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.755952 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0906f4\"" Oct 29 22:58:46.962051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.952334 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4rk\" (UniqueName: \"kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:46.962051 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:46.952508 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:47.069047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.067919 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:47.069047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.068218 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4rk\" (UniqueName: \"kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:47.076740 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.069456 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:47.140438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.139326 5367 generic.go:356] "Generic (PLEG): container finished" podID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerID="46151889d69c1730d46096ef7b7a59ded5dc61f1e58ceabde89e075efead0f8a" exitCode=0 Oct 29 22:58:47.140438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.139539 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" event={"ID":"8b18a7d5-0996-425b-96ce-54196e33acbd","Type":"ContainerDied","Data":"46151889d69c1730d46096ef7b7a59ded5dc61f1e58ceabde89e075efead0f8a"} Oct 29 22:58:47.140438 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.139632 5367 scope.go:122] "RemoveContainer" containerID="e67e1bbc9330a8b70f06d7491f94b3d651ed7434bbe32dc382f8bd74249c20af" Oct 29 22:58:47.151792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.149144 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4rk\" (UniqueName: \"kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:47.350787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.338756 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:47.546161 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.546111 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.618935 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619194 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619236 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619316 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619352 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619390 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619448 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619504 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.619844 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.619583 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy\") pod \"8b18a7d5-0996-425b-96ce-54196e33acbd\" (UID: \"8b18a7d5-0996-425b-96ce-54196e33acbd\") " Oct 29 22:58:47.671729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.671165 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:47.673216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.673122 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.673216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.673189 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.673621 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.673603 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.700410 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.700011 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.703665 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.703617 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "kube-api-access-t7x4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:47.704011 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.703973 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736253 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736292 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7x4g\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-kube-api-access-t7x4g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736308 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736322 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736336 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b18a7d5-0996-425b-96ce-54196e33acbd-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736349 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.742747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.736363 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.802404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.802348 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:58:47.814793 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.807767 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy" pod "8b18a7d5-0996-425b-96ce-54196e33acbd" (UID: "8b18a7d5-0996-425b-96ce-54196e33acbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.838036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.837996 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.838036 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:47.838034 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b18a7d5-0996-425b-96ce-54196e33acbd-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:58:48.010305 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:48.010274 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2194a3_766d_48a4_8c3a_2eccc7e417ab.slice/crio-e1439111ba1d223391d883dec06bfec7d564c069678381c222556026329db3c1 WatchSource:0}: Error finding container e1439111ba1d223391d883dec06bfec7d564c069678381c222556026329db3c1: Status 404 returned error can't find the container with id e1439111ba1d223391d883dec06bfec7d564c069678381c222556026329db3c1 Oct 29 22:58:48.011183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.011158 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd"] Oct 29 22:58:48.069178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.069152 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2"] Oct 29 22:58:48.077404 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.077378 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5e10ef-bw6p2"] Oct 29 22:58:48.149490 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.149453 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" event={"ID":"5f2194a3-766d-48a4-8c3a-2eccc7e417ab","Type":"ContainerStarted","Data":"e1439111ba1d223391d883dec06bfec7d564c069678381c222556026329db3c1"} Oct 29 22:58:48.150977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.150956 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" Oct 29 22:58:48.151103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.150985 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp" event={"ID":"8b18a7d5-0996-425b-96ce-54196e33acbd","Type":"ContainerDied","Data":"87fad629d60c811c89b0ef783bfdc4ee3cabd583b0986a596abc4ef75fb31dea"} Oct 29 22:58:48.151103 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.151031 5367 scope.go:122] "RemoveContainer" containerID="46151889d69c1730d46096ef7b7a59ded5dc61f1e58ceabde89e075efead0f8a" Oct 29 22:58:48.210898 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.210866 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp"] Oct 29 22:58:48.224294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:48.224266 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-09fc89-openshift-default-869878cbf5-sgssp"] Oct 29 22:58:49.195164 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:49.195121 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" event={"ID":"5f2194a3-766d-48a4-8c3a-2eccc7e417ab","Type":"ContainerStarted","Data":"9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca"} Oct 29 22:58:49.196450 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:49.196420 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:58:49.225941 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:49.225906 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0" path="/var/lib/kubelet/pods/7dcf36b9-4090-4b5d-a0c6-dbab8dd2aec0/volumes" Oct 29 22:58:49.226601 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:49.226584 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" path="/var/lib/kubelet/pods/8b18a7d5-0996-425b-96ce-54196e33acbd/volumes" Oct 29 22:58:49.241003 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:49.240956 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" podStartSLOduration=3.240943652 podStartE2EDuration="3.240943652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:49.239249315 +0000 UTC m=+2596.133324281" watchObservedRunningTime="2026-10-29 22:58:49.240943652 +0000 UTC m=+2596.135018574" Oct 29 22:58:52.332717 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:58:52.332668 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:58:58.292334 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.292245 5367 generic.go:356] "Generic (PLEG): container finished" podID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerID="a8843d17970f09b33187e5ad49adef68078296495d75b14496fd54494052073f" exitCode=0 Oct 29 22:58:58.292888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.292325 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" event={"ID":"933caef7-daeb-4332-97a6-5ea2aedfe708","Type":"ContainerDied","Data":"a8843d17970f09b33187e5ad49adef68078296495d75b14496fd54494052073f"} Oct 29 22:58:58.292959 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.292925 5367 scope.go:122] "RemoveContainer" containerID="a8843d17970f09b33187e5ad49adef68078296495d75b14496fd54494052073f" Oct 29 22:58:58.294884 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.294856 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:58.294963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.294926 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:58.294963 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.294957 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:58.443099 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.443062 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s"] Oct 29 22:58:58.444315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.444286 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" Oct 29 22:58:58.444315 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.444312 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" Oct 29 22:58:58.444579 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.444563 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" Oct 29 22:58:58.444647 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.444594 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" Oct 29 22:58:58.448560 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.448534 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.475792 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.474276 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-14b0c3-openshift-default-dockercfg-7qggr\"" Oct 29 22:58:58.478875 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.478849 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s"] Oct 29 22:58:58.574909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.574879 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575060 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575220 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575195 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575263 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575245 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575351 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575335 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575396 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575380 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575462 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575413 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248q4\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575494 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.575580 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.575556 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.684745 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.684958 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685067 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685103 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685163 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685214 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248q4\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685257 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.685723 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685310 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.686403 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.685919 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.697724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.693255 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.697724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.693509 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.701722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.698738 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.701722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.699031 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.735718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.732026 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.755722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.754596 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.777277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.769372 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:58.777277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:58.770285 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248q4\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4\") pod \"e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:59.067337 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.067309 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:58:59.303610 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.303572 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" event={"ID":"933caef7-daeb-4332-97a6-5ea2aedfe708","Type":"ContainerStarted","Data":"76c303c76ad842c38f7ca1ed060220693bf67078c58aa9dc72f4d08d31a1a95e"} Oct 29 22:58:59.397106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.397013 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s"] Oct 29 22:58:59.407875 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:58:59.407841 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43abfea5_a446_4ccf_be1b_b0dbcc19a091.slice/crio-6f59ec2a6c65b746a29a18b200b907578b0d71d83d880cd689861e99310194bf WatchSource:0}: Error finding container 6f59ec2a6c65b746a29a18b200b907578b0d71d83d880cd689861e99310194bf: Status 404 returned error can't find the container with id 6f59ec2a6c65b746a29a18b200b907578b0d71d83d880cd689861e99310194bf Oct 29 22:58:59.411318 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.411273 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:59.411436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.411371 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:59.411436 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.411417 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:59.640746 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.640713 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:58:59.827479 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.827448 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:58:59.829171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.829145 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 29 22:58:59.829353 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:58:59.829192 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 29 22:59:00.314407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:00.314367 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" event={"ID":"43abfea5-a446-4ccf-be1b-b0dbcc19a091","Type":"ContainerStarted","Data":"d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543"} Oct 29 22:59:00.314888 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:00.314436 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" event={"ID":"43abfea5-a446-4ccf-be1b-b0dbcc19a091","Type":"ContainerStarted","Data":"6f59ec2a6c65b746a29a18b200b907578b0d71d83d880cd689861e99310194bf"} Oct 29 22:59:00.391299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:00.391250 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podStartSLOduration=2.391236286 podStartE2EDuration="2.391236286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:00.375181764 +0000 UTC m=+2607.269256688" watchObservedRunningTime="2026-10-29 22:59:00.391236286 +0000 UTC m=+2607.285311256" Oct 29 22:59:00.827197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:00.827162 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 29 22:59:00.827387 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:00.827219 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 29 22:59:01.079171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.078925 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:59:01.085562 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.085533 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:59:01.360776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.351978 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 22:59:01.572913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.571789 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:59:01.572913 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.571902 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 22:59:01.819250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.819213 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x"] Oct 29 22:59:01.827724 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.825901 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c7b96a-kzv5x"] Oct 29 22:59:01.842644 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.842544 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 29 22:59:01.843001 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:01.842968 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 29 22:59:02.827110 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:02.827072 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 29 22:59:02.827649 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:02.827137 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 29 22:59:03.224453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:03.224369 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7782b4-7a51-4e8b-b966-101bae1cb930" path="/var/lib/kubelet/pods/dd7782b4-7a51-4e8b-b966-101bae1cb930/volumes" Oct 29 22:59:03.827233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:03.827201 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 29 22:59:03.827756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:03.827725 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 29 22:59:04.827360 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:04.827320 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 29 22:59:04.827855 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:04.827376 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 29 22:59:05.978407 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:05.978376 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:59:05.980336 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:05.980308 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 22:59:10.138324 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.138286 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88"] Oct 29 22:59:10.139261 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.139240 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" Oct 29 22:59:10.139382 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.139263 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b18a7d5-0996-425b-96ce-54196e33acbd" containerName="istio-proxy" Oct 29 22:59:10.143375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.143352 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.168348 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.168307 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88"] Oct 29 22:59:10.170767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.170734 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:10.170909 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.170784 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:59:10.336729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.323991 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.336729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.324097 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.336729 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.324165 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkm4\" (UniqueName: \"kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.425574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.425482 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.425574 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.425550 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.425839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.425593 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkm4\" (UniqueName: \"kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.426319 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.426293 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.427978 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.427958 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.445767 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.445675 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkm4\" (UniqueName: \"kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.462668 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.462646 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:10.728270 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.728203 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv"] Oct 29 22:59:10.732923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.732904 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.761306 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.761274 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv"] Oct 29 22:59:10.815735 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.814051 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88"] Oct 29 22:59:10.832284 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832254 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832377 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832489 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832474 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832498 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832528 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhk2\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832657 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832568 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832766 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832739 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832869 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832850 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.832917 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.832897 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934310 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934377 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934475 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934496 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934554 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhk2\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934647 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934860 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934771 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934893 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.934971 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934927 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.935084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934971 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.935084 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.934889 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.935200 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.935149 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.935259 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.935209 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.935299 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.935279 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.936947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.936922 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.937294 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.937277 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.955825 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.955794 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:10.957521 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:10.957496 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhk2\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:11.049041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.048962 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:11.372539 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.372512 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv"] Oct 29 22:59:11.372539 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:59:11.372530 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730090a7_005a_414c_9764_1313ac482dfc.slice/crio-aabfad1fd48b0cbbd4d33b0064155b81b88166f56e24bebe96a681511441533e WatchSource:0}: Error finding container aabfad1fd48b0cbbd4d33b0064155b81b88166f56e24bebe96a681511441533e: Status 404 returned error can't find the container with id aabfad1fd48b0cbbd4d33b0064155b81b88166f56e24bebe96a681511441533e Oct 29 22:59:11.375548 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.375508 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:59:11.375669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.375594 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:59:11.375669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.375642 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:59:11.431540 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.431506 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk"] Oct 29 22:59:11.437788 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.437765 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.456039 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.456014 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e803b6-openshift-default-dockercfg-q4w9g\"" Oct 29 22:59:11.467332 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.467301 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk"] Oct 29 22:59:11.543553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.543512 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.543787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.543631 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.543901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.543880 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.544016 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.543997 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.544228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.544205 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.544288 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.544260 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.544368 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.544349 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qjz\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.544428 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.544384 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.544532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.544512 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.646570 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.646460 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.646772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.646633 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.646772 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.646761 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.646932 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.646912 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.646998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.646964 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647027 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qjz\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647060 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647053 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647073 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647099 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647129 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647150 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647534 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647511 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647787 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647763 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.647921 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.647895 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.649444 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.649422 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.649655 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.649639 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.669640 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.669606 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qjz\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.669995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.669976 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.702633 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.702598 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" event={"ID":"65ade42f-618e-438c-ba76-17e4c70582b4","Type":"ContainerStarted","Data":"be0e806d0f9002e90f9d0229d02119303ae75b69d3c24054afef37965dac7152"} Oct 29 22:59:11.702815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.702655 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" event={"ID":"65ade42f-618e-438c-ba76-17e4c70582b4","Type":"ContainerStarted","Data":"d22f1f111e3951a22f39f1eb793bd16e9dc7ba1640d408d37683a93bdbb3b042"} Oct 29 22:59:11.704853 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.704828 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" event={"ID":"730090a7-005a-414c-9764-1313ac482dfc","Type":"ContainerStarted","Data":"45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de"} Oct 29 22:59:11.704995 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.704868 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" event={"ID":"730090a7-005a-414c-9764-1313ac482dfc","Type":"ContainerStarted","Data":"aabfad1fd48b0cbbd4d33b0064155b81b88166f56e24bebe96a681511441533e"} Oct 29 22:59:11.747249 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.747197 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" podStartSLOduration=1.747181318 podStartE2EDuration="1.747181318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:11.745795041 +0000 UTC m=+2618.639869960" watchObservedRunningTime="2026-10-29 22:59:11.747181318 +0000 UTC m=+2618.641256239" Oct 29 22:59:11.774557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.774517 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:11.793768 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:11.793684 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podStartSLOduration=1.7936658159999999 podStartE2EDuration="1.793665816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:11.791438893 +0000 UTC m=+2618.685513815" watchObservedRunningTime="2026-10-29 22:59:11.793665816 +0000 UTC m=+2618.687740737" Oct 29 22:59:12.049684 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.049613 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:12.051295 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.051268 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:12.051422 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.051317 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:12.109292 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:59:12.109260 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86fe3ea_1ba9_4b52_9020_6207ebb7bebd.slice/crio-32f994acd72e4548d4b5ba3f502a91d28248f52cf20026e34bdff34c4d714743 WatchSource:0}: Error finding container 32f994acd72e4548d4b5ba3f502a91d28248f52cf20026e34bdff34c4d714743: Status 404 returned error can't find the container with id 32f994acd72e4548d4b5ba3f502a91d28248f52cf20026e34bdff34c4d714743 Oct 29 22:59:12.109583 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.109560 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk"] Oct 29 22:59:12.112194 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.112159 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:59:12.112307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.112238 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:59:12.112307 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.112286 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:59:12.715820 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.715723 5367 generic.go:356] "Generic (PLEG): container finished" podID="65ade42f-618e-438c-ba76-17e4c70582b4" containerID="be0e806d0f9002e90f9d0229d02119303ae75b69d3c24054afef37965dac7152" exitCode=0 Oct 29 22:59:12.716258 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.715819 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" event={"ID":"65ade42f-618e-438c-ba76-17e4c70582b4","Type":"ContainerDied","Data":"be0e806d0f9002e90f9d0229d02119303ae75b69d3c24054afef37965dac7152"} Oct 29 22:59:12.717626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.717599 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" event={"ID":"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd","Type":"ContainerStarted","Data":"8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501"} Oct 29 22:59:12.717738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.717655 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" event={"ID":"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd","Type":"ContainerStarted","Data":"32f994acd72e4548d4b5ba3f502a91d28248f52cf20026e34bdff34c4d714743"} Oct 29 22:59:12.789762 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.789003 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:12.843312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:12.843226 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podStartSLOduration=1.843206323 podStartE2EDuration="1.843206323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:12.838974492 +0000 UTC m=+2619.733049423" watchObservedRunningTime="2026-10-29 22:59:12.843206323 +0000 UTC m=+2619.737281246" Oct 29 22:59:13.049390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:13.049310 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:13.049390 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:13.049366 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:13.789987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:13.789614 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:59:13.789987 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:13.789677 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:59:14.075827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.050286 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:14.075827 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.050350 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:14.113376 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.113346 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:14.117994 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.117969 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4\" (UniqueName: \"kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4\") pod \"65ade42f-618e-438c-ba76-17e4c70582b4\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " Oct 29 22:59:14.118365 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.118347 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca\") pod \"65ade42f-618e-438c-ba76-17e4c70582b4\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " Oct 29 22:59:14.118569 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.118554 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca\") pod \"65ade42f-618e-438c-ba76-17e4c70582b4\" (UID: \"65ade42f-618e-438c-ba76-17e4c70582b4\") " Oct 29 22:59:14.118861 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.118842 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca" pod "65ade42f-618e-438c-ba76-17e4c70582b4" (UID: "65ade42f-618e-438c-ba76-17e4c70582b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:14.119123 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.119106 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:14.120198 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.120175 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4" pod "65ade42f-618e-438c-ba76-17e4c70582b4" (UID: "65ade42f-618e-438c-ba76-17e4c70582b4"). InnerVolumeSpecName "kube-api-access-dxkm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:14.120468 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.120447 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca" pod "65ade42f-618e-438c-ba76-17e4c70582b4" (UID: "65ade42f-618e-438c-ba76-17e4c70582b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:14.250771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.248958 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ade42f-618e-438c-ba76-17e4c70582b4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:14.250771 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.248990 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkm4\" (UniqueName: \"kubernetes.io/projected/65ade42f-618e-438c-ba76-17e4c70582b4-kube-api-access-dxkm4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:14.738281 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.738253 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" Oct 29 22:59:14.738487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.738284 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88" event={"ID":"65ade42f-618e-438c-ba76-17e4c70582b4","Type":"ContainerDied","Data":"d22f1f111e3951a22f39f1eb793bd16e9dc7ba1640d408d37683a93bdbb3b042"} Oct 29 22:59:14.738487 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.738332 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22f1f111e3951a22f39f1eb793bd16e9dc7ba1640d408d37683a93bdbb3b042" Oct 29 22:59:14.780722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.779944 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:59:14.780722 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:14.780004 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:59:15.050155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:15.050051 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:15.050155 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:15.050116 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:15.784840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:15.784797 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:59:15.785147 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:15.785117 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:59:16.051758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:16.051666 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:16.051758 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:16.051737 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:16.789867 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:16.789829 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:59:16.790166 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:16.790137 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:59:17.049675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:17.049592 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:17.049675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:17.049646 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:17.784350 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:17.784314 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:59:17.785000 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:17.784955 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:59:18.049923 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:18.049839 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 29 22:59:18.050186 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:18.050157 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 29 22:59:18.778152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:18.778116 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:59:18.778478 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:18.778432 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:59:19.621848 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:19.621816 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:19.623216 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:19.623179 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:19.624391 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:19.624354 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 22:59:19.797641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:19.797608 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:19.798191 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:19.798167 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:19.803576 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:19.803524 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 22:59:20.929573 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:20.929539 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb"] Oct 29 22:59:20.930120 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:20.929753 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" podUID="da53d7be-436c-4373-9572-4fa61531175d" containerName="maas-api" containerID="cri-o://41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190" gracePeriod=30 Oct 29 22:59:21.210192 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.210167 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:59:21.346278 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.346238 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs\") pod \"da53d7be-436c-4373-9572-4fa61531175d\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " Oct 29 22:59:21.354718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.346725 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca\") pod \"da53d7be-436c-4373-9572-4fa61531175d\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " Oct 29 22:59:21.376721 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.355184 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp\" (UniqueName: \"kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp\") pod \"da53d7be-436c-4373-9572-4fa61531175d\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " Oct 29 22:59:21.376953 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.364392 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca" pod "da53d7be-436c-4373-9572-4fa61531175d" (UID: "da53d7be-436c-4373-9572-4fa61531175d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:21.377087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.377038 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls\") pod \"da53d7be-436c-4373-9572-4fa61531175d\" (UID: \"da53d7be-436c-4373-9572-4fa61531175d\") " Oct 29 22:59:21.414734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.378055 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da53d7be-436c-4373-9572-4fa61531175d-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:21.457776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.426893 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls" pod "da53d7be-436c-4373-9572-4fa61531175d" (UID: "da53d7be-436c-4373-9572-4fa61531175d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:21.457776 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.446461 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp" pod "da53d7be-436c-4373-9572-4fa61531175d" (UID: "da53d7be-436c-4373-9572-4fa61531175d"). InnerVolumeSpecName "kube-api-access-hpggp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:21.480067 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.469736 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs" pod "da53d7be-436c-4373-9572-4fa61531175d" (UID: "da53d7be-436c-4373-9572-4fa61531175d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:21.520287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.520191 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:21.520287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.520226 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpggp\" (UniqueName: \"kubernetes.io/projected/da53d7be-436c-4373-9572-4fa61531175d-kube-api-access-hpggp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:21.520287 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.520241 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da53d7be-436c-4373-9572-4fa61531175d-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:21.888217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.888187 5367 generic.go:356] "Generic (PLEG): container finished" podID="da53d7be-436c-4373-9572-4fa61531175d" containerID="41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190" exitCode=0 Oct 29 22:59:21.888217 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.888230 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" Oct 29 22:59:21.888453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.888274 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" event={"ID":"da53d7be-436c-4373-9572-4fa61531175d","Type":"ContainerDied","Data":"41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190"} Oct 29 22:59:21.888453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.888333 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb" event={"ID":"da53d7be-436c-4373-9572-4fa61531175d","Type":"ContainerDied","Data":"9ae7b1444d430ba9efc27c6a7992d92802e0e7165e9e0d429fd468a460a9fefc"} Oct 29 22:59:21.888453 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.888360 5367 scope.go:122] "RemoveContainer" containerID="41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190" Oct 29 22:59:21.901205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.901190 5367 scope.go:122] "RemoveContainer" containerID="41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190" Oct 29 22:59:21.901517 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:59:21.901498 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190\": container with ID starting with 41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190 not found: ID does not exist" containerID="41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190" Oct 29 22:59:21.901602 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.901528 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190"} err="failed to get container status \"41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190\": rpc error: code = NotFound desc = could not find container \"41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190\": container with ID starting with 41ffc82233fa596a7afa47abbf0f79ebe3e00ecbc6c0793be3fe5a5af03f2190 not found: ID does not exist" Oct 29 22:59:21.948277 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.948224 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb"] Oct 29 22:59:21.959550 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:21.959521 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e526ec-7bd98b976b-gwxtb"] Oct 29 22:59:22.193488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:22.193372 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88"] Oct 29 22:59:22.240049 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:22.230604 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e526ec-5vj88"] Oct 29 22:59:22.332509 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:59:22.332465 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:59:23.224682 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:23.224647 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ade42f-618e-438c-ba76-17e4c70582b4" path="/var/lib/kubelet/pods/65ade42f-618e-438c-ba76-17e4c70582b4/volumes" Oct 29 22:59:23.225183 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:23.225113 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da53d7be-436c-4373-9572-4fa61531175d" path="/var/lib/kubelet/pods/da53d7be-436c-4373-9572-4fa61531175d/volumes" Oct 29 22:59:29.029747 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:29.029714 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk"] Oct 29 22:59:29.042451 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:29.042417 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df93b9bf-lrggk"] Oct 29 22:59:29.223998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:29.223965 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2248a7-ed0d-404f-b10a-20573d112291" path="/var/lib/kubelet/pods/3d2248a7-ed0d-404f-b10a-20573d112291/volumes" Oct 29 22:59:30.181141 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.181107 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf"] Oct 29 22:59:30.181890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.181874 5367 cpu_manager.go:414] "containerMap: removing container" podUID="da53d7be-436c-4373-9572-4fa61531175d" containerName="maas-api" Oct 29 22:59:30.181954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.181891 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da53d7be-436c-4373-9572-4fa61531175d" containerName="maas-api" Oct 29 22:59:30.181954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.181903 5367 cpu_manager.go:414] "containerMap: removing container" podUID="65ade42f-618e-438c-ba76-17e4c70582b4" containerName="revoke-keys" Oct 29 22:59:30.181954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.181909 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ade42f-618e-438c-ba76-17e4c70582b4" containerName="revoke-keys" Oct 29 22:59:30.182055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.182031 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="da53d7be-436c-4373-9572-4fa61531175d" containerName="maas-api" Oct 29 22:59:30.182055 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.182043 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ade42f-618e-438c-ba76-17e4c70582b4" containerName="revoke-keys" Oct 29 22:59:30.187459 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.187440 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.206664 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.206640 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf"] Oct 29 22:59:30.214197 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.214166 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:30.214872 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.214852 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:59:30.215087 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.215063 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:59:30.219375 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.219356 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:59:30.256890 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.256864 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.257218 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.257195 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.257311 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.257246 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5d6\" (UniqueName: \"kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.369718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.361803 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.369718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.361859 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5d6\" (UniqueName: \"kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.369718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.361959 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.369718 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.363482 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.379728 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.378496 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.395894 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.391923 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5d6\" (UniqueName: \"kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6\") pod \"maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.502517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.502418 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:30.811896 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:59:30.811856 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537fe2b0_7515_4242_bb6f_fedd9d53f2a3.slice/crio-825456b0cb92091ebf1b79546d62741ccf6b6ccabf69fc50f763bb2571e18892 WatchSource:0}: Error finding container 825456b0cb92091ebf1b79546d62741ccf6b6ccabf69fc50f763bb2571e18892: Status 404 returned error can't find the container with id 825456b0cb92091ebf1b79546d62741ccf6b6ccabf69fc50f763bb2571e18892 Oct 29 22:59:30.812543 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.812521 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf"] Oct 29 22:59:30.975250 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:30.975223 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" event={"ID":"537fe2b0-7515-4242-bb6f-fedd9d53f2a3","Type":"ContainerStarted","Data":"825456b0cb92091ebf1b79546d62741ccf6b6ccabf69fc50f763bb2571e18892"} Oct 29 22:59:31.986658 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:31.986624 5367 generic.go:356] "Generic (PLEG): container finished" podID="537fe2b0-7515-4242-bb6f-fedd9d53f2a3" containerID="5083dfe177f5b000d8554131dead5580149e76b643451b7221f61d8a5bf5cdab" exitCode=0 Oct 29 22:59:31.987112 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:31.986663 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" event={"ID":"537fe2b0-7515-4242-bb6f-fedd9d53f2a3","Type":"ContainerDied","Data":"5083dfe177f5b000d8554131dead5580149e76b643451b7221f61d8a5bf5cdab"} Oct 29 22:59:33.156182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.156158 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:33.219312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.219276 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6\" (UniqueName: \"kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6\") pod \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " Oct 29 22:59:33.219498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.219366 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca\") pod \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " Oct 29 22:59:33.219498 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.219433 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca\") pod \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\" (UID: \"537fe2b0-7515-4242-bb6f-fedd9d53f2a3\") " Oct 29 22:59:33.220565 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.220535 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca" pod "537fe2b0-7515-4242-bb6f-fedd9d53f2a3" (UID: "537fe2b0-7515-4242-bb6f-fedd9d53f2a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:33.223111 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.223085 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6" pod "537fe2b0-7515-4242-bb6f-fedd9d53f2a3" (UID: "537fe2b0-7515-4242-bb6f-fedd9d53f2a3"). InnerVolumeSpecName "kube-api-access-lb5d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:33.236975 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.236949 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca" pod "537fe2b0-7515-4242-bb6f-fedd9d53f2a3" (UID: "537fe2b0-7515-4242-bb6f-fedd9d53f2a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:33.321271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.321244 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb5d6\" (UniqueName: \"kubernetes.io/projected/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-kube-api-access-lb5d6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:33.321271 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.321268 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:33.321467 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:33.321278 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537fe2b0-7515-4242-bb6f-fedd9d53f2a3-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:34.005504 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:34.005476 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" Oct 29 22:59:34.005675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:34.005513 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf" event={"ID":"537fe2b0-7515-4242-bb6f-fedd9d53f2a3","Type":"ContainerDied","Data":"825456b0cb92091ebf1b79546d62741ccf6b6ccabf69fc50f763bb2571e18892"} Oct 29 22:59:34.005675 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:34.005564 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="825456b0cb92091ebf1b79546d62741ccf6b6ccabf69fc50f763bb2571e18892" Oct 29 22:59:34.975358 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:34.975322 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl"] Oct 29 22:59:34.994732 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:34.994682 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh"] Oct 29 22:59:38.290237 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.290195 5367 scope.go:122] "RemoveContainer" containerID="74f86ffb55d86efc7021779e9379570fd2ac2c53fd8c5bf002dd7b3ec99aa187" Oct 29 22:59:38.304868 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.304847 5367 scope.go:122] "RemoveContainer" containerID="21d1b38735e6a49b52be630f6280cd4ebe1b83d2989b8e0f89cc4355af661e33" Oct 29 22:59:38.327373 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.327345 5367 scope.go:122] "RemoveContainer" containerID="d1ea17afdb066064f7b03222a4893d155e8d23cceaaa01ce4f18a8d1b77242ce" Oct 29 22:59:38.337510 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.337488 5367 scope.go:122] "RemoveContainer" containerID="7bfd5bbfadcc951aaf47a224769777f3e154a01ffa98600fa4b3bdb23de3a9ad" Oct 29 22:59:38.347182 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.347160 5367 scope.go:122] "RemoveContainer" containerID="0d1e24638524afa6e06af28f0810210979d2e9b7e48538064b20b271c89c8906" Oct 29 22:59:38.357205 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.357180 5367 scope.go:122] "RemoveContainer" containerID="a497abc73e48192671aaadfd53b7306e1ff1879424ed3086ad9d3509abcb1a9d" Oct 29 22:59:38.367608 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:38.367585 5367 scope.go:122] "RemoveContainer" containerID="be4fcd96e2574d75a031804befacbeaf49ec93b38e7d77733d403f1819c2fb33" Oct 29 22:59:40.660085 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:40.660046 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp"] Oct 29 22:59:43.031530 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:43.031489 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf"] Oct 29 22:59:43.043669 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:43.043638 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c633eaa9-5cwwf"] Oct 29 22:59:43.232174 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:43.232138 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31ddf96-84fc-4d1a-95e9-0f363ad410a9" path="/var/lib/kubelet/pods/b31ddf96-84fc-4d1a-95e9-0f363ad410a9/volumes" Oct 29 22:59:45.660546 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:45.660489 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" containerName="payload-pre-processing" containerID="cri-o://56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7" gracePeriod=25 Oct 29 22:59:47.488286 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:47.488245 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:59:50.028603 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.028555 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" containerID="cri-o://0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd" gracePeriod=45 Oct 29 22:59:50.054783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.054109 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" containerID="cri-o://ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395" gracePeriod=45 Oct 29 22:59:50.211730 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.210155 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": dial tcp 10.133.0.117:8000: connect: connection refused" Oct 29 22:59:50.516666 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.516636 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:59:50.531999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.531752 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:59:50.557240 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.557204 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home\") pod \"1b7fd0ba-3560-45c9-b059-679e64e807a5\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " Oct 29 22:59:50.557509 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.557495 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm\") pod \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " Oct 29 22:59:50.557632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.557622 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g\" (UniqueName: \"kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g\") pod \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " Oct 29 22:59:50.557839 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.557823 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs\") pod \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " Oct 29 22:59:50.557998 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.557981 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q\" (UniqueName: \"kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q\") pod \"1b7fd0ba-3560-45c9-b059-679e64e807a5\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " Oct 29 22:59:50.558171 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.558158 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs\") pod \"1b7fd0ba-3560-45c9-b059-679e64e807a5\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " Oct 29 22:59:50.558275 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.558262 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache\") pod \"1b7fd0ba-3560-45c9-b059-679e64e807a5\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " Oct 29 22:59:50.558454 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.558432 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir\") pod \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " Oct 29 22:59:50.558677 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.558663 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache\") pod \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " Oct 29 22:59:50.558832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.558818 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm\") pod \"1b7fd0ba-3560-45c9-b059-679e64e807a5\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " Oct 29 22:59:50.558947 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.558935 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir\") pod \"1b7fd0ba-3560-45c9-b059-679e64e807a5\" (UID: \"1b7fd0ba-3560-45c9-b059-679e64e807a5\") " Oct 29 22:59:50.559061 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.559048 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home\") pod \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\" (UID: \"0eb12b49-8fc5-4d29-8dd3-8016cc31162f\") " Oct 29 22:59:50.560398 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.560374 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home" pod "0eb12b49-8fc5-4d29-8dd3-8016cc31162f" (UID: "0eb12b49-8fc5-4d29-8dd3-8016cc31162f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.560840 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.560818 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home" pod "1b7fd0ba-3560-45c9-b059-679e64e807a5" (UID: "1b7fd0ba-3560-45c9-b059-679e64e807a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.588641 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.588584 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache" pod "1b7fd0ba-3560-45c9-b059-679e64e807a5" (UID: "1b7fd0ba-3560-45c9-b059-679e64e807a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.596517 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.596459 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir" pod "0eb12b49-8fc5-4d29-8dd3-8016cc31162f" (UID: "0eb12b49-8fc5-4d29-8dd3-8016cc31162f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.596851 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.596824 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache" pod "0eb12b49-8fc5-4d29-8dd3-8016cc31162f" (UID: "0eb12b49-8fc5-4d29-8dd3-8016cc31162f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.597988 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.597784 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm" pod "0eb12b49-8fc5-4d29-8dd3-8016cc31162f" (UID: "0eb12b49-8fc5-4d29-8dd3-8016cc31162f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.598531 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.598154 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir" pod "1b7fd0ba-3560-45c9-b059-679e64e807a5" (UID: "1b7fd0ba-3560-45c9-b059-679e64e807a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.617132 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.617082 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g" pod "0eb12b49-8fc5-4d29-8dd3-8016cc31162f" (UID: "0eb12b49-8fc5-4d29-8dd3-8016cc31162f"). InnerVolumeSpecName "kube-api-access-p6p2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:50.634247 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.634217 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q" pod "1b7fd0ba-3560-45c9-b059-679e64e807a5" (UID: "1b7fd0ba-3560-45c9-b059-679e64e807a5"). InnerVolumeSpecName "kube-api-access-92h9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:50.634427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.634217 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm" pod "1b7fd0ba-3560-45c9-b059-679e64e807a5" (UID: "1b7fd0ba-3560-45c9-b059-679e64e807a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.637753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.634288 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs" pod "1b7fd0ba-3560-45c9-b059-679e64e807a5" (UID: "1b7fd0ba-3560-45c9-b059-679e64e807a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:50.637753 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.634345 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs" pod "0eb12b49-8fc5-4d29-8dd3-8016cc31162f" (UID: "0eb12b49-8fc5-4d29-8dd3-8016cc31162f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.672983 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673055 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6p2g\" (UniqueName: \"kubernetes.io/projected/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-kube-api-access-p6p2g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673072 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673089 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92h9q\" (UniqueName: \"kubernetes.io/projected/1b7fd0ba-3560-45c9-b059-679e64e807a5-kube-api-access-92h9q\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673101 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7fd0ba-3560-45c9-b059-679e64e807a5-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673115 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673129 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673141 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673153 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673164 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673177 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb12b49-8fc5-4d29-8dd3-8016cc31162f-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.673233 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:50.673188 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7fd0ba-3560-45c9-b059-679e64e807a5-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:51.167752 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.167454 5367 generic.go:356] "Generic (PLEG): container finished" podID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerID="ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395" exitCode=0 Oct 29 22:59:51.168296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.167766 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" event={"ID":"1b7fd0ba-3560-45c9-b059-679e64e807a5","Type":"ContainerDied","Data":"ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395"} Oct 29 22:59:51.168296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.167797 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" event={"ID":"1b7fd0ba-3560-45c9-b059-679e64e807a5","Type":"ContainerDied","Data":"1ecaf3222ee16de93079e6e363a2364c4d1a85161abd34aa1d8d4ebde8258ce3"} Oct 29 22:59:51.168296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.167827 5367 scope.go:122] "RemoveContainer" containerID="ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395" Oct 29 22:59:51.168296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.167993 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh" Oct 29 22:59:51.171480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.171311 5367 generic.go:356] "Generic (PLEG): container finished" podID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerID="0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd" exitCode=0 Oct 29 22:59:51.171480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.171438 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" Oct 29 22:59:51.171480 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.171452 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" event={"ID":"0eb12b49-8fc5-4d29-8dd3-8016cc31162f","Type":"ContainerDied","Data":"0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd"} Oct 29 22:59:51.171688 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.171483 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl" event={"ID":"0eb12b49-8fc5-4d29-8dd3-8016cc31162f","Type":"ContainerDied","Data":"fed73fa13506aa3ff56afdb742a2a7810d9d92e94c876c1dfd4d4eda8f2d0534"} Oct 29 22:59:51.187427 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.187402 5367 scope.go:122] "RemoveContainer" containerID="ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395" Oct 29 22:59:51.187852 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:59:51.187826 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395\": container with ID starting with ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395 not found: ID does not exist" containerID="ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395" Oct 29 22:59:51.187968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.187858 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395"} err="failed to get container status \"ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395\": rpc error: code = NotFound desc = could not find container \"ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395\": container with ID starting with ac19f44a7c9ebedebb5cdf7618e3ae4a594df2b4718d3bf69a14e812974bd395 not found: ID does not exist" Oct 29 22:59:51.187968 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.187876 5367 scope.go:122] "RemoveContainer" containerID="0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd" Oct 29 22:59:51.198904 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.198881 5367 scope.go:122] "RemoveContainer" containerID="0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd" Oct 29 22:59:51.199212 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:59:51.199191 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd\": container with ID starting with 0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd not found: ID does not exist" containerID="0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd" Oct 29 22:59:51.199298 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.199222 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd"} err="failed to get container status \"0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd\": rpc error: code = NotFound desc = could not find container \"0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd\": container with ID starting with 0a5a943517f7595692208ac0017c61386f8f3740d14fb12e0dfa38990975fbcd not found: ID does not exist" Oct 29 22:59:51.248012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.247971 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh"] Oct 29 22:59:51.258557 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.258515 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/premium-w0-40cf9a-kserve-bcdb45979-5wrbh"] Oct 29 22:59:51.317815 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.317774 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl"] Oct 29 22:59:51.329877 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:51.329801 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-2-w0-40cf9a-kserve-575f48b6d5-kmlsl"] Oct 29 22:59:52.340993 ip-10-0-139-29 kubenswrapper[5367]: E1029 22:59:52.340950 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 22:59:53.233626 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:53.233595 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" path="/var/lib/kubelet/pods/0eb12b49-8fc5-4d29-8dd3-8016cc31162f/volumes" Oct 29 22:59:53.234094 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:53.234078 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" path="/var/lib/kubelet/pods/1b7fd0ba-3560-45c9-b059-679e64e807a5/volumes" Oct 29 22:59:54.037808 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.037770 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr"] Oct 29 22:59:54.038847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.038814 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" Oct 29 22:59:54.038847 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.038834 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" Oct 29 22:59:54.038977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.038905 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" Oct 29 22:59:54.038977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.038913 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" Oct 29 22:59:54.038977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.038931 5367 cpu_manager.go:414] "containerMap: removing container" podUID="537fe2b0-7515-4242-bb6f-fedd9d53f2a3" containerName="revoke-keys" Oct 29 22:59:54.038977 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.038939 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537fe2b0-7515-4242-bb6f-fedd9d53f2a3" containerName="revoke-keys" Oct 29 22:59:54.039122 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.039115 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7fd0ba-3560-45c9-b059-679e64e807a5" containerName="main" Oct 29 22:59:54.039158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.039138 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb12b49-8fc5-4d29-8dd3-8016cc31162f" containerName="main" Oct 29 22:59:54.039158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.039147 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="537fe2b0-7515-4242-bb6f-fedd9d53f2a3" containerName="revoke-keys" Oct 29 22:59:54.045228 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.045207 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.067678 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.067647 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr"] Oct 29 22:59:54.074400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.074373 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:59:54.074400 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.074389 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:54.094541 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.094513 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:59:54.163012 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.162980 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:59:54.181522 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.181493 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.181715 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.181667 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c84m\" (UniqueName: \"kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.181930 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.181912 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.258553 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.258502 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9"] Oct 29 22:59:54.263129 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.263112 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.283756 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.283657 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.283915 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.283857 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.284296 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.284019 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c84m\" (UniqueName: \"kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.284843 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.284816 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.285558 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.285529 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9"] Oct 29 22:59:54.286876 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.286851 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.310047 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.310008 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c84m\" (UniqueName: \"kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m\") pod \"maas-api-revoke-sub-00c7700b-6z7lr\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.360178 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.360139 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:54.385842 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.385805 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.385986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.385856 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9mj\" (UniqueName: \"kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.386139 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.386123 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.488393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.487935 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.488393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.488066 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.488393 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.488101 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9mj\" (UniqueName: \"kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.489599 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.489349 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.502149 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.502078 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.512571 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.511836 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9mj\" (UniqueName: \"kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj\") pod \"maas-api-revoke-sub-115d83cc-fbll9\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.577938 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.577862 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:54.677999 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.677337 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr"] Oct 29 22:59:54.679730 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:59:54.679675 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f49f61_a708_4b72_bb22_09b6662b7979.slice/crio-9182df0f20f2e39b97a93f00205974e353206d79cecceb39a2e1a680570b64b2 WatchSource:0}: Error finding container 9182df0f20f2e39b97a93f00205974e353206d79cecceb39a2e1a680570b64b2: Status 404 returned error can't find the container with id 9182df0f20f2e39b97a93f00205974e353206d79cecceb39a2e1a680570b64b2 Oct 29 22:59:54.742108 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.742079 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z"] Oct 29 22:59:54.748138 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.748115 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.766076 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.766048 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z"] Oct 29 22:59:54.884364 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:59:54.884334 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69f26a2_cc87_409b_9a95_0b2f44910015.slice/crio-136476c952c6d4012995a8cf3efd38747b4899f353c0fcd0976210e1636de11c WatchSource:0}: Error finding container 136476c952c6d4012995a8cf3efd38747b4899f353c0fcd0976210e1636de11c: Status 404 returned error can't find the container with id 136476c952c6d4012995a8cf3efd38747b4899f353c0fcd0976210e1636de11c Oct 29 22:59:54.885273 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.885247 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9"] Oct 29 22:59:54.894031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.894007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84tp\" (UniqueName: \"kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.894180 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.894139 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.894255 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.894239 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.995803 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.995775 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.995986 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.995845 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.996041 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.996015 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84tp\" (UniqueName: \"kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.996532 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.996509 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:54.998106 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:54.998082 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:55.016738 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.016713 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84tp\" (UniqueName: \"kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp\") pod \"maas-api-revoke-sub-ef262b67-qlf6z\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:55.062819 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.062794 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:55.302303 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.302230 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" event={"ID":"e69f26a2-cc87-409b-9a95-0b2f44910015","Type":"ContainerStarted","Data":"136476c952c6d4012995a8cf3efd38747b4899f353c0fcd0976210e1636de11c"} Oct 29 22:59:55.304312 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.304280 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" event={"ID":"93f49f61-a708-4b72-bb22-09b6662b7979","Type":"ContainerStarted","Data":"b1767dc328770d4b7edc7e3b50e2b1ff87a9e398d986688f6b8d879209381563"} Oct 29 22:59:55.304456 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.304327 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" event={"ID":"93f49f61-a708-4b72-bb22-09b6662b7979","Type":"ContainerStarted","Data":"9182df0f20f2e39b97a93f00205974e353206d79cecceb39a2e1a680570b64b2"} Oct 29 22:59:55.352488 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.352438 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" podStartSLOduration=1.352423703 podStartE2EDuration="1.352423703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:55.347142433 +0000 UTC m=+2662.241217355" watchObservedRunningTime="2026-10-29 22:59:55.352423703 +0000 UTC m=+2662.246498647" Oct 29 22:59:55.386336 ip-10-0-139-29 kubenswrapper[5367]: W1029 22:59:55.386301 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d5a7bd_378b_4f2e_b596_eed156b70bb0.slice/crio-4ba392bcddea81c454299f08421fbb12175ad18a31c3de8dead7432dcf242dc1 WatchSource:0}: Error finding container 4ba392bcddea81c454299f08421fbb12175ad18a31c3de8dead7432dcf242dc1: Status 404 returned error can't find the container with id 4ba392bcddea81c454299f08421fbb12175ad18a31c3de8dead7432dcf242dc1 Oct 29 22:59:55.387170 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:55.387148 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z"] Oct 29 22:59:56.037891 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.037810 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44"] Oct 29 22:59:56.042304 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.042276 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8918ccd-hqj44"] Oct 29 22:59:56.314632 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.314595 5367 generic.go:356] "Generic (PLEG): container finished" podID="93f49f61-a708-4b72-bb22-09b6662b7979" containerID="b1767dc328770d4b7edc7e3b50e2b1ff87a9e398d986688f6b8d879209381563" exitCode=0 Oct 29 22:59:56.315130 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.314666 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" event={"ID":"93f49f61-a708-4b72-bb22-09b6662b7979","Type":"ContainerDied","Data":"b1767dc328770d4b7edc7e3b50e2b1ff87a9e398d986688f6b8d879209381563"} Oct 29 22:59:56.316832 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.316811 5367 generic.go:356] "Generic (PLEG): container finished" podID="e69f26a2-cc87-409b-9a95-0b2f44910015" containerID="a7d0144c89412dba50fc3285b890125cafe788ee74a59bccb3eaa56505132e6d" exitCode=0 Oct 29 22:59:56.316955 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.316897 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" event={"ID":"e69f26a2-cc87-409b-9a95-0b2f44910015","Type":"ContainerDied","Data":"a7d0144c89412dba50fc3285b890125cafe788ee74a59bccb3eaa56505132e6d"} Oct 29 22:59:56.318799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.318778 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" event={"ID":"31d5a7bd-378b-4f2e-b596-eed156b70bb0","Type":"ContainerStarted","Data":"5d0711f218ed572dec339d12e0e74a448f093b7aecd824602723f9a291eead47"} Oct 29 22:59:56.318935 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.318817 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" event={"ID":"31d5a7bd-378b-4f2e-b596-eed156b70bb0","Type":"ContainerStarted","Data":"4ba392bcddea81c454299f08421fbb12175ad18a31c3de8dead7432dcf242dc1"} Oct 29 22:59:56.442945 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:56.442894 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" podStartSLOduration=2.442880593 podStartE2EDuration="2.442880593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:56.440479141 +0000 UTC m=+2663.334554066" watchObservedRunningTime="2026-10-29 22:59:56.442880593 +0000 UTC m=+2663.336955513" Oct 29 22:59:57.224004 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.223968 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282335fe-80d1-4c66-8025-837a636e1026" path="/var/lib/kubelet/pods/282335fe-80d1-4c66-8025-837a636e1026/volumes" Oct 29 22:59:57.328152 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.328118 5367 generic.go:356] "Generic (PLEG): container finished" podID="31d5a7bd-378b-4f2e-b596-eed156b70bb0" containerID="5d0711f218ed572dec339d12e0e74a448f093b7aecd824602723f9a291eead47" exitCode=0 Oct 29 22:59:57.328567 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.328194 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" event={"ID":"31d5a7bd-378b-4f2e-b596-eed156b70bb0","Type":"ContainerDied","Data":"5d0711f218ed572dec339d12e0e74a448f093b7aecd824602723f9a291eead47"} Oct 29 22:59:57.488783 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.488671 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:59:57.565148 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.565118 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:57.568069 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.568048 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:57.635515 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.635477 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m\" (UniqueName: \"kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m\") pod \"93f49f61-a708-4b72-bb22-09b6662b7979\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " Oct 29 22:59:57.635751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.635525 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca\") pod \"e69f26a2-cc87-409b-9a95-0b2f44910015\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " Oct 29 22:59:57.635751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.635571 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca\") pod \"e69f26a2-cc87-409b-9a95-0b2f44910015\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " Oct 29 22:59:57.635751 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.635727 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca\") pod \"93f49f61-a708-4b72-bb22-09b6662b7979\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " Oct 29 22:59:57.635958 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.635787 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj\" (UniqueName: \"kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj\") pod \"e69f26a2-cc87-409b-9a95-0b2f44910015\" (UID: \"e69f26a2-cc87-409b-9a95-0b2f44910015\") " Oct 29 22:59:57.636093 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.636065 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca\") pod \"93f49f61-a708-4b72-bb22-09b6662b7979\" (UID: \"93f49f61-a708-4b72-bb22-09b6662b7979\") " Oct 29 22:59:57.636502 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.636473 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca" pod "93f49f61-a708-4b72-bb22-09b6662b7979" (UID: "93f49f61-a708-4b72-bb22-09b6662b7979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:57.636901 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.636879 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:57.638381 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.638359 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m" pod "93f49f61-a708-4b72-bb22-09b6662b7979" (UID: "93f49f61-a708-4b72-bb22-09b6662b7979"). InnerVolumeSpecName "kube-api-access-7c84m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:57.638491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.638379 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca" pod "93f49f61-a708-4b72-bb22-09b6662b7979" (UID: "93f49f61-a708-4b72-bb22-09b6662b7979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:57.638491 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.638435 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca" pod "e69f26a2-cc87-409b-9a95-0b2f44910015" (UID: "e69f26a2-cc87-409b-9a95-0b2f44910015"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:57.638871 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.638845 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj" pod "e69f26a2-cc87-409b-9a95-0b2f44910015" (UID: "e69f26a2-cc87-409b-9a95-0b2f44910015"). InnerVolumeSpecName "kube-api-access-rj9mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:57.663158 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.663117 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca" pod "e69f26a2-cc87-409b-9a95-0b2f44910015" (UID: "e69f26a2-cc87-409b-9a95-0b2f44910015"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:57.738031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.737992 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c84m\" (UniqueName: \"kubernetes.io/projected/93f49f61-a708-4b72-bb22-09b6662b7979-kube-api-access-7c84m\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:57.738031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.738021 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:57.738031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.738031 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69f26a2-cc87-409b-9a95-0b2f44910015-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:57.738031 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.738039 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f49f61-a708-4b72-bb22-09b6662b7979-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:57.738395 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:57.738048 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj9mj\" (UniqueName: \"kubernetes.io/projected/e69f26a2-cc87-409b-9a95-0b2f44910015-kube-api-access-rj9mj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:58.032460 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.032424 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7"] Oct 29 22:59:58.036799 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.036768 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1528505d-lrzr7"] Oct 29 22:59:58.343323 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.343298 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" Oct 29 22:59:58.343838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.343290 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9" event={"ID":"e69f26a2-cc87-409b-9a95-0b2f44910015","Type":"ContainerDied","Data":"136476c952c6d4012995a8cf3efd38747b4899f353c0fcd0976210e1636de11c"} Oct 29 22:59:58.343838 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.343420 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="136476c952c6d4012995a8cf3efd38747b4899f353c0fcd0976210e1636de11c" Oct 29 22:59:58.344973 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.344949 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" event={"ID":"93f49f61-a708-4b72-bb22-09b6662b7979","Type":"ContainerDied","Data":"9182df0f20f2e39b97a93f00205974e353206d79cecceb39a2e1a680570b64b2"} Oct 29 22:59:58.345089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.344999 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9182df0f20f2e39b97a93f00205974e353206d79cecceb39a2e1a680570b64b2" Oct 29 22:59:58.345089 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.344961 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr" Oct 29 22:59:58.512008 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.511983 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:58.651846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.651760 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp\" (UniqueName: \"kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp\") pod \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " Oct 29 22:59:58.651846 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.651818 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca\") pod \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " Oct 29 22:59:58.652091 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.651894 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca\") pod \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\" (UID: \"31d5a7bd-378b-4f2e-b596-eed156b70bb0\") " Oct 29 22:59:58.652954 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.652924 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca" pod "31d5a7bd-378b-4f2e-b596-eed156b70bb0" (UID: "31d5a7bd-378b-4f2e-b596-eed156b70bb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:58.654612 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.654566 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca" pod "31d5a7bd-378b-4f2e-b596-eed156b70bb0" (UID: "31d5a7bd-378b-4f2e-b596-eed156b70bb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:58.655796 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.655770 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp" pod "31d5a7bd-378b-4f2e-b596-eed156b70bb0" (UID: "31d5a7bd-378b-4f2e-b596-eed156b70bb0"). InnerVolumeSpecName "kube-api-access-l84tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:58.754356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.754325 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84tp\" (UniqueName: \"kubernetes.io/projected/31d5a7bd-378b-4f2e-b596-eed156b70bb0-kube-api-access-l84tp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:58.754356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.754352 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:58.754356 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:58.754362 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5a7bd-378b-4f2e-b596-eed156b70bb0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 22:59:59.224734 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:59.224673 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6b166a-67a6-48aa-9c8a-3f744836e3c9" path="/var/lib/kubelet/pods/4e6b166a-67a6-48aa-9c8a-3f744836e3c9/volumes" Oct 29 22:59:59.354206 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:59.354176 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" Oct 29 22:59:59.354624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:59.354201 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z" event={"ID":"31d5a7bd-378b-4f2e-b596-eed156b70bb0","Type":"ContainerDied","Data":"4ba392bcddea81c454299f08421fbb12175ad18a31c3de8dead7432dcf242dc1"} Oct 29 22:59:59.354624 ip-10-0-139-29 kubenswrapper[5367]: I1029 22:59:59.354252 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba392bcddea81c454299f08421fbb12175ad18a31c3de8dead7432dcf242dc1" Oct 29 23:00:00.156489 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.156454 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m"] Oct 29 23:00:00.157283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157265 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e69f26a2-cc87-409b-9a95-0b2f44910015" containerName="revoke-keys" Oct 29 23:00:00.157348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157285 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69f26a2-cc87-409b-9a95-0b2f44910015" containerName="revoke-keys" Oct 29 23:00:00.157348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157301 5367 cpu_manager.go:414] "containerMap: removing container" podUID="93f49f61-a708-4b72-bb22-09b6662b7979" containerName="revoke-keys" Oct 29 23:00:00.157348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157306 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f49f61-a708-4b72-bb22-09b6662b7979" containerName="revoke-keys" Oct 29 23:00:00.157348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157315 5367 cpu_manager.go:414] "containerMap: removing container" podUID="31d5a7bd-378b-4f2e-b596-eed156b70bb0" containerName="revoke-keys" Oct 29 23:00:00.157348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157321 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d5a7bd-378b-4f2e-b596-eed156b70bb0" containerName="revoke-keys" Oct 29 23:00:00.157504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157420 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69f26a2-cc87-409b-9a95-0b2f44910015" containerName="revoke-keys" Oct 29 23:00:00.157504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157428 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f49f61-a708-4b72-bb22-09b6662b7979" containerName="revoke-keys" Oct 29 23:00:00.157504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.157436 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d5a7bd-378b-4f2e-b596-eed156b70bb0" containerName="revoke-keys" Oct 29 23:00:00.170472 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.170432 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp"] Oct 29 23:00:00.175393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.175361 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.179728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.178319 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn"] Oct 29 23:00:00.179728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.178730 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.187092 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.184572 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm"] Oct 29 23:00:00.187092 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.184671 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.192312 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.192290 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.192534 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.192511 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95"] Oct 29 23:00:00.198193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.198168 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.200369 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.200346 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp"] Oct 29 23:00:00.215766 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.211548 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm"] Oct 29 23:00:00.224536 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.224503 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:00:00.224663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.224564 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:00:00.224663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.224634 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:00:00.229430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.229405 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m"] Oct 29 23:00:00.229559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.229498 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95"] Oct 29 23:00:00.229628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.229601 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.237404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.237370 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk"] Oct 29 23:00:00.246953 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.244108 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.246953 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.244205 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq"] Oct 29 23:00:00.252553 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.252521 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.273015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.270777 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb"] Oct 29 23:00:00.320574 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.315970 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.324630 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.324751 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.324842 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.324884 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzt8b\" (UniqueName: \"kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.324934 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpj5\" (UniqueName: \"kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.325524 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.325568 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.325631 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7vm\" (UniqueName: \"kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.325664 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k84z\" (UniqueName: \"kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.325768 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929qk\" (UniqueName: \"kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.325799 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8fj\" (UniqueName: \"kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.326267 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrtr\" (UniqueName: \"kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.326350 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.326429 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.329479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.327380 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n"] Oct 29 23:00:00.335438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.334406 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.340413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.340349 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm"] Oct 29 23:00:00.350010 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.349367 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn"] Oct 29 23:00:00.367001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.363366 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk"] Oct 29 23:00:00.375590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.372254 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq"] Oct 29 23:00:00.383763 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.380230 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n"] Oct 29 23:00:00.393666 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.388779 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb"] Oct 29 23:00:00.409236 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.408754 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj"] Oct 29 23:00:00.414327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.414300 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.416914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.416327 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp"] Oct 29 23:00:00.423973 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.423944 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.428094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428067 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm"] Oct 29 23:00:00.428242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428095 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njsbq\" (UniqueName: \"kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.428242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428166 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.428242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428211 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7zq\" (UniqueName: \"kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.428427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428398 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzt8b\" (UniqueName: \"kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.428481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428455 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpj5\" (UniqueName: \"kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.428543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428514 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.428602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428549 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.428656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428611 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ddp\" (UniqueName: \"kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.428798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428777 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.428925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428818 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.428925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428869 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7vm\" (UniqueName: \"kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.428925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428897 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k84z\" (UniqueName: \"kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.428896 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429014 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.429095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429073 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929qk\" (UniqueName: \"kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.429257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429102 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8fj\" (UniqueName: \"kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.429257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429144 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrtr\" (UniqueName: \"kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.429257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429199 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.429425 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429272 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.429425 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429338 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.429425 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.429394 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.430104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.430079 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.430203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.430129 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.430608 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.430287 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.430836 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.430813 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.430928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.430872 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.431726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.431670 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.435259 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.435234 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5"] Oct 29 23:00:00.442791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.442757 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj"] Oct 29 23:00:00.442932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.442902 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.447395 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.447369 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp"] Oct 29 23:00:00.452626 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.452593 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzt8b\" (UniqueName: \"kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.455215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.455191 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k84z\" (UniqueName: \"kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.455333 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.455239 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm"] Oct 29 23:00:00.459518 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.459487 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8fj\" (UniqueName: \"kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.459617 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.459521 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrtr\" (UniqueName: \"kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr\") pod \"maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.459617 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.459530 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-929qk\" (UniqueName: \"kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.460461 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.460424 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7vm\" (UniqueName: \"kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.462054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.462030 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpj5\" (UniqueName: \"kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.463356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.463336 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56"] Oct 29 23:00:00.463547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.463440 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.477007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.476102 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.486249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.486204 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5"] Oct 29 23:00:00.491581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.491555 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.493348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.493330 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm"] Oct 29 23:00:00.496599 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.496580 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:00.501022 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.501004 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56"] Oct 29 23:00:00.506563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.506543 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7"] Oct 29 23:00:00.511320 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.511296 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5"] Oct 29 23:00:00.511411 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.511398 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.515745 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.515680 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5"] Oct 29 23:00:00.521200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.521180 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:00.524492 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.524473 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7"] Oct 29 23:00:00.531436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.531408 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.531748 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.531678 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzv5\" (UniqueName: \"kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.531910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.531858 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njsbq\" (UniqueName: \"kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.531910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.531890 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.532093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.531933 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.532093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.531973 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7zq\" (UniqueName: \"kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.532093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.532055 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.532093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.532086 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.532398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.532114 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxv56\" (UniqueName: \"kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.532398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.532152 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66grn\" (UniqueName: \"kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.532398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.532189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ddp\" (UniqueName: \"kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.532398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.532376 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.533227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.533140 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.533413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.533300 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.534872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.534844 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.544866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.544844 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:00.566961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.566930 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:00.568625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.568571 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ddp\" (UniqueName: \"kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.578819 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.574001 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njsbq\" (UniqueName: \"kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.581020 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.580990 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7zq\" (UniqueName: \"kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.618122 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.617864 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:00.628522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.627954 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:00.634643 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.633785 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.634643 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.633899 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.634643 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634026 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2rw\" (UniqueName: \"kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.634643 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634071 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzv5\" (UniqueName: \"kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.634643 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634101 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.634945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634155 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.634945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634872 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.634945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634937 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.634984 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.635073 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.635121 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmr2\" (UniqueName: \"kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2\") pod \"maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.635189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxv56\" (UniqueName: \"kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.635242 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdfd\" (UniqueName: \"kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd\") pod \"maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.635277 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66grn\" (UniqueName: \"kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.636415 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.636494 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.640559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.640425 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.665399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.665175 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:00.665760 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.665687 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66grn\" (UniqueName: \"kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.678005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.677943 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxv56\" (UniqueName: \"kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56\") pod \"maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.680601 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.680397 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:00.687194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.686992 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:00.689342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.689316 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzv5\" (UniqueName: \"kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.690817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.690793 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:00.737156 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.737115 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:00.739571 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.738736 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2rw\" (UniqueName: \"kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.739571 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.738791 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.740681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.740635 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.740898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.738835 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.741065 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.741018 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.741728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.741611 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.741728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.741672 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmr2\" (UniqueName: \"kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2\") pod \"maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.743155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.743119 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.744691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.743304 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdfd\" (UniqueName: \"kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd\") pod \"maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.744691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.743460 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.744691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.744277 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.744691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.744641 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:00.747871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.747820 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.766817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.764014 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2rw\" (UniqueName: \"kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.779748 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.779663 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.781601 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.781551 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmr2\" (UniqueName: \"kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2\") pod \"maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.784224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.784186 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdfd\" (UniqueName: \"kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd\") pod \"maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:00.805968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.805721 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:00.887911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.887551 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:00.917228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.917158 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:00.953064 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.952842 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:00.954920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:00.954866 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:01.228211 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.228175 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8003be9d_b7c7_4b0a_b988_02d13f22102b.slice/crio-c750b2af1fbf1f30cca331d0b4ad3f360fc6c46739e681b3f3a2282ddf3a114d WatchSource:0}: Error finding container c750b2af1fbf1f30cca331d0b4ad3f360fc6c46739e681b3f3a2282ddf3a114d: Status 404 returned error can't find the container with id c750b2af1fbf1f30cca331d0b4ad3f360fc6c46739e681b3f3a2282ddf3a114d Oct 29 23:00:01.231363 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.231343 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m"] Oct 29 23:00:01.267465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.267437 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp"] Oct 29 23:00:01.288444 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.288412 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722b2177_f08d_4203_b365_310ef1c795a5.slice/crio-0dca4bf66350183e9f3bac67f348cb6f910513933f3af4a03d2ecc8a67674cf7 WatchSource:0}: Error finding container 0dca4bf66350183e9f3bac67f348cb6f910513933f3af4a03d2ecc8a67674cf7: Status 404 returned error can't find the container with id 0dca4bf66350183e9f3bac67f348cb6f910513933f3af4a03d2ecc8a67674cf7 Oct 29 23:00:01.322862 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.322329 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64ffbae_aa0d_4478_8a93_07c0094d7c63.slice/crio-4de4d60d695978dc8ed21b9fffd692eb5f38b51ceeb2afa9ee8c2f0550a1e4a3 WatchSource:0}: Error finding container 4de4d60d695978dc8ed21b9fffd692eb5f38b51ceeb2afa9ee8c2f0550a1e4a3: Status 404 returned error can't find the container with id 4de4d60d695978dc8ed21b9fffd692eb5f38b51ceeb2afa9ee8c2f0550a1e4a3 Oct 29 23:00:01.327567 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.327542 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn"] Oct 29 23:00:01.383618 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.383584 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerStarted","Data":"c750b2af1fbf1f30cca331d0b4ad3f360fc6c46739e681b3f3a2282ddf3a114d"} Oct 29 23:00:01.385281 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.385253 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerStarted","Data":"4de4d60d695978dc8ed21b9fffd692eb5f38b51ceeb2afa9ee8c2f0550a1e4a3"} Oct 29 23:00:01.386588 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.386562 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerStarted","Data":"0dca4bf66350183e9f3bac67f348cb6f910513933f3af4a03d2ecc8a67674cf7"} Oct 29 23:00:01.421831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.421803 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm"] Oct 29 23:00:01.424238 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.424205 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee465e5_6f19_4d52_9e85_00e40b15bd06.slice/crio-cdf26b0e32700293167463217c7d093911f6dca5eaa2d58b7bf48b7f4a3e09e8 WatchSource:0}: Error finding container cdf26b0e32700293167463217c7d093911f6dca5eaa2d58b7bf48b7f4a3e09e8: Status 404 returned error can't find the container with id cdf26b0e32700293167463217c7d093911f6dca5eaa2d58b7bf48b7f4a3e09e8 Oct 29 23:00:01.436932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.433502 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95"] Oct 29 23:00:01.479190 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.479156 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq"] Oct 29 23:00:01.558825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.552159 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56"] Oct 29 23:00:01.562424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.561824 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n"] Oct 29 23:00:01.568296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.568268 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm"] Oct 29 23:00:01.579930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.579898 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb"] Oct 29 23:00:01.584468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.584441 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk"] Oct 29 23:00:01.589530 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.589500 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5"] Oct 29 23:00:01.613571 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.613544 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7"] Oct 29 23:00:01.629030 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.629003 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm"] Oct 29 23:00:01.638815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.638783 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp"] Oct 29 23:00:01.645159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.645136 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj"] Oct 29 23:00:01.660823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:01.660799 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5"] Oct 29 23:00:01.664304 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.664280 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c78c9_5de2_4bc5_91e5_9f891eb598b4.slice/crio-c45ccec23d4da46e9d9c833cf3589856d4c9edb915e675aa016cbb7c2eb56962 WatchSource:0}: Error finding container c45ccec23d4da46e9d9c833cf3589856d4c9edb915e675aa016cbb7c2eb56962: Status 404 returned error can't find the container with id c45ccec23d4da46e9d9c833cf3589856d4c9edb915e675aa016cbb7c2eb56962 Oct 29 23:00:01.760404 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.760322 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ba151d_e166_4c58_8d4e_6e507a0986e1.slice/crio-081727edfb397d37b08051f9b17d03e2f3480b02fc01324eae4cf3890f3f9d11 WatchSource:0}: Error finding container 081727edfb397d37b08051f9b17d03e2f3480b02fc01324eae4cf3890f3f9d11: Status 404 returned error can't find the container with id 081727edfb397d37b08051f9b17d03e2f3480b02fc01324eae4cf3890f3f9d11 Oct 29 23:00:01.761373 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.761348 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81935cce_192e_4569_afe7_b61fe0a9a419.slice/crio-df0f2fdebfc25c65cf530209760aafa49a343522e44c9bdbeebd171495e06e71 WatchSource:0}: Error finding container df0f2fdebfc25c65cf530209760aafa49a343522e44c9bdbeebd171495e06e71: Status 404 returned error can't find the container with id df0f2fdebfc25c65cf530209760aafa49a343522e44c9bdbeebd171495e06e71 Oct 29 23:00:01.764175 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.764155 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdf2542_f96c_4c37_88e7_fdb0a7778b43.slice/crio-7b791374e729d328135689af9dfcb41e8c088157c1da1b13d6b8f56c178959de WatchSource:0}: Error finding container 7b791374e729d328135689af9dfcb41e8c088157c1da1b13d6b8f56c178959de: Status 404 returned error can't find the container with id 7b791374e729d328135689af9dfcb41e8c088157c1da1b13d6b8f56c178959de Oct 29 23:00:01.857349 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.857314 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd119d843_8ea0_444b_9590_614da9a4f435.slice/crio-01dd03301dc721c541a0f4d23521197c62fc8c009f69e7d1c7bee1e8993e1040 WatchSource:0}: Error finding container 01dd03301dc721c541a0f4d23521197c62fc8c009f69e7d1c7bee1e8993e1040: Status 404 returned error can't find the container with id 01dd03301dc721c541a0f4d23521197c62fc8c009f69e7d1c7bee1e8993e1040 Oct 29 23:00:01.858294 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.858192 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f056935_3821_4d4a_8fe6_7a9e41a9c6eb.slice/crio-f027b1b596032790e678fcad745025a3d6dd87a47f5c705fb5c55232cb41674a WatchSource:0}: Error finding container f027b1b596032790e678fcad745025a3d6dd87a47f5c705fb5c55232cb41674a: Status 404 returned error can't find the container with id f027b1b596032790e678fcad745025a3d6dd87a47f5c705fb5c55232cb41674a Oct 29 23:00:01.860909 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.860887 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7531bdb4_381a_4039_bcea_5ed7379a8546.slice/crio-12e6d0c0e9472a8755c3275351133c44d2ceaf0ffbb05ca65bc72f0e25c60f77 WatchSource:0}: Error finding container 12e6d0c0e9472a8755c3275351133c44d2ceaf0ffbb05ca65bc72f0e25c60f77: Status 404 returned error can't find the container with id 12e6d0c0e9472a8755c3275351133c44d2ceaf0ffbb05ca65bc72f0e25c60f77 Oct 29 23:00:01.862245 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.862225 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef8b4aa_4cbf_43d5_ac35_43a87301a566.slice/crio-ac31a9d783c9ed017f428b6e14c74059d9aa09236c17550123b1cb831863c63d WatchSource:0}: Error finding container ac31a9d783c9ed017f428b6e14c74059d9aa09236c17550123b1cb831863c63d: Status 404 returned error can't find the container with id ac31a9d783c9ed017f428b6e14c74059d9aa09236c17550123b1cb831863c63d Oct 29 23:00:01.864601 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:01.864455 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba7d901_6282_41be_ad0f_fd74055bad81.slice/crio-78f39c382863d5cd4046327ecebbc698ffafcbfbe037961c70306dd10b038bd8 WatchSource:0}: Error finding container 78f39c382863d5cd4046327ecebbc698ffafcbfbe037961c70306dd10b038bd8: Status 404 returned error can't find the container with id 78f39c382863d5cd4046327ecebbc698ffafcbfbe037961c70306dd10b038bd8 Oct 29 23:00:02.396964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.396930 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerStarted","Data":"5eb02b7d190f217988258f79e053d5a0671323bd37587ec3bb96ac0c0329b482"} Oct 29 23:00:02.397401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.396984 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerStarted","Data":"f9d775f3ce66ad738959e9d53747da3cbdc4c6e383ea7ad80bfc5dbc264248f6"} Oct 29 23:00:02.398814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.398786 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerStarted","Data":"b7f6be789e988ff899331700196401c5b288e51542ac7c96779644592a9eba20"} Oct 29 23:00:02.399014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.398839 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerStarted","Data":"2191c882a5d3210d2c1af70a48c766dba189e867c3331ba4e51fe7fb36c65ada"} Oct 29 23:00:02.400269 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.400249 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerStarted","Data":"ac31a9d783c9ed017f428b6e14c74059d9aa09236c17550123b1cb831863c63d"} Oct 29 23:00:02.401810 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.401792 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerStarted","Data":"b3f742901d93046fae67e24900e88a74ae7316c6ff03d497553a6182da3cc0c4"} Oct 29 23:00:02.403163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.403143 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerStarted","Data":"f027b1b596032790e678fcad745025a3d6dd87a47f5c705fb5c55232cb41674a"} Oct 29 23:00:02.404372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.404353 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerStarted","Data":"c45ccec23d4da46e9d9c833cf3589856d4c9edb915e675aa016cbb7c2eb56962"} Oct 29 23:00:02.405917 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.405894 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerStarted","Data":"b6832e89b46d0df3f18703c64dc7b0e87f05326bceef823c0d8b70ad703eb2a2"} Oct 29 23:00:02.407193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.407174 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerStarted","Data":"fd8305d3c4e1778356ef363ba34e867be48727b41123fac7188013023b82e011"} Oct 29 23:00:02.408455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.408436 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerStarted","Data":"12e6d0c0e9472a8755c3275351133c44d2ceaf0ffbb05ca65bc72f0e25c60f77"} Oct 29 23:00:02.409988 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.409967 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerStarted","Data":"acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7"} Oct 29 23:00:02.410074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.410004 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerStarted","Data":"cdf26b0e32700293167463217c7d093911f6dca5eaa2d58b7bf48b7f4a3e09e8"} Oct 29 23:00:02.411252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.411231 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerStarted","Data":"01dd03301dc721c541a0f4d23521197c62fc8c009f69e7d1c7bee1e8993e1040"} Oct 29 23:00:02.412481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.412463 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerStarted","Data":"78f39c382863d5cd4046327ecebbc698ffafcbfbe037961c70306dd10b038bd8"} Oct 29 23:00:02.414021 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.413998 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerStarted","Data":"f461b696336a5add6695675a55b730168f4fce62ee091dde0da0dfdd7b6900f0"} Oct 29 23:00:02.415842 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.415816 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerStarted","Data":"7b791374e729d328135689af9dfcb41e8c088157c1da1b13d6b8f56c178959de"} Oct 29 23:00:02.417261 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.417238 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerStarted","Data":"df0f2fdebfc25c65cf530209760aafa49a343522e44c9bdbeebd171495e06e71"} Oct 29 23:00:02.418474 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.418446 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerStarted","Data":"081727edfb397d37b08051f9b17d03e2f3480b02fc01324eae4cf3890f3f9d11"} Oct 29 23:00:02.419745 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.419726 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerStarted","Data":"95037ca50ad23d11c9da0bfc6e7b6807df52f971319490494c774eaf6b13501d"} Oct 29 23:00:02.445785 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.445746 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" podStartSLOduration=2.445732581 podStartE2EDuration="2.445732581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.440735412 +0000 UTC m=+2669.334810332" watchObservedRunningTime="2026-10-29 23:00:02.445732581 +0000 UTC m=+2669.339807507" Oct 29 23:00:02.485418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.485367 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" podStartSLOduration=2.485352951 podStartE2EDuration="2.485352951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.483017163 +0000 UTC m=+2669.377092085" watchObservedRunningTime="2026-10-29 23:00:02.485352951 +0000 UTC m=+2669.379427870" Oct 29 23:00:02.528743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.528677 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" podStartSLOduration=2.528663108 podStartE2EDuration="2.528663108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.523534268 +0000 UTC m=+2669.417609191" watchObservedRunningTime="2026-10-29 23:00:02.528663108 +0000 UTC m=+2669.422738084" Oct 29 23:00:02.569881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.569835 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" podStartSLOduration=2.569820289 podStartE2EDuration="2.569820289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.563435603 +0000 UTC m=+2669.457510526" watchObservedRunningTime="2026-10-29 23:00:02.569820289 +0000 UTC m=+2669.463895209" Oct 29 23:00:02.605416 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.605372 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" podStartSLOduration=2.605357675 podStartE2EDuration="2.605357675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.604690658 +0000 UTC m=+2669.498765577" watchObservedRunningTime="2026-10-29 23:00:02.605357675 +0000 UTC m=+2669.499432594" Oct 29 23:00:02.652541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:02.652489 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" podStartSLOduration=2.65247581 podStartE2EDuration="2.65247581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.646786075 +0000 UTC m=+2669.540860997" watchObservedRunningTime="2026-10-29 23:00:02.65247581 +0000 UTC m=+2669.546550740" Oct 29 23:00:02.951056 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:02.951019 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee465e5_6f19_4d52_9e85_00e40b15bd06.slice/crio-acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee465e5_6f19_4d52_9e85_00e40b15bd06.slice/crio-conmon-acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:00:03.431581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.431480 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerStarted","Data":"a5886dbe1009ba84c712627ab6d8966400bc20bae7ff5337bd17b931148195ed"} Oct 29 23:00:03.433287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.433255 5367 generic.go:356] "Generic (PLEG): container finished" podID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerID="b3f742901d93046fae67e24900e88a74ae7316c6ff03d497553a6182da3cc0c4" exitCode=60 Oct 29 23:00:03.433413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.433323 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerDied","Data":"b3f742901d93046fae67e24900e88a74ae7316c6ff03d497553a6182da3cc0c4"} Oct 29 23:00:03.433632 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.433611 5367 scope.go:122] "RemoveContainer" containerID="b3f742901d93046fae67e24900e88a74ae7316c6ff03d497553a6182da3cc0c4" Oct 29 23:00:03.435118 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.435093 5367 generic.go:356] "Generic (PLEG): container finished" podID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerID="08cc882e1eaae1dae9a5fb683b879bf690c655267421b1ac9712a5b046c6d5a3" exitCode=60 Oct 29 23:00:03.435228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.435159 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerDied","Data":"08cc882e1eaae1dae9a5fb683b879bf690c655267421b1ac9712a5b046c6d5a3"} Oct 29 23:00:03.435418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.435367 5367 scope.go:122] "RemoveContainer" containerID="08cc882e1eaae1dae9a5fb683b879bf690c655267421b1ac9712a5b046c6d5a3" Oct 29 23:00:03.437093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.437075 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerStarted","Data":"c944b7832c9cd2ea0298ed60ad3227aae982c4b56a19606ce54e72d56c30e37d"} Oct 29 23:00:03.438867 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.438849 5367 generic.go:356] "Generic (PLEG): container finished" podID="722b2177-f08d-4203-b365-310ef1c795a5" containerID="b6832e89b46d0df3f18703c64dc7b0e87f05326bceef823c0d8b70ad703eb2a2" exitCode=60 Oct 29 23:00:03.438946 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.438918 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerDied","Data":"b6832e89b46d0df3f18703c64dc7b0e87f05326bceef823c0d8b70ad703eb2a2"} Oct 29 23:00:03.439139 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.439092 5367 scope.go:122] "RemoveContainer" containerID="b6832e89b46d0df3f18703c64dc7b0e87f05326bceef823c0d8b70ad703eb2a2" Oct 29 23:00:03.440580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.440558 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerStarted","Data":"fb2d1fa08ca4e7ec6e0d6bc7a81c6439dd5152915785081477ec2f9ad289a19e"} Oct 29 23:00:03.442251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.442202 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerStarted","Data":"ae8225938def214593173ce6e7f48328cef759dc8c6f2cc720af47012ae08838"} Oct 29 23:00:03.443896 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.443878 5367 generic.go:356] "Generic (PLEG): container finished" podID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerID="acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7" exitCode=60 Oct 29 23:00:03.443978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.443960 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerDied","Data":"acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7"} Oct 29 23:00:03.444167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.444150 5367 scope.go:122] "RemoveContainer" containerID="acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7" Oct 29 23:00:03.445562 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.445537 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerStarted","Data":"c7e447c24777ee271e5f4ae559fffe469b49a6c6ad0e6982768b955aed3ef9aa"} Oct 29 23:00:03.447273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.447249 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerStarted","Data":"32f94865974c413d10f8605e90ff9f9d97b1e7b6af01faaf93186380ce719a8b"} Oct 29 23:00:03.449222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.449201 5367 generic.go:356] "Generic (PLEG): container finished" podID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerID="f461b696336a5add6695675a55b730168f4fce62ee091dde0da0dfdd7b6900f0" exitCode=60 Oct 29 23:00:03.449327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.449281 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerDied","Data":"f461b696336a5add6695675a55b730168f4fce62ee091dde0da0dfdd7b6900f0"} Oct 29 23:00:03.449471 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.449454 5367 scope.go:122] "RemoveContainer" containerID="f461b696336a5add6695675a55b730168f4fce62ee091dde0da0dfdd7b6900f0" Oct 29 23:00:03.451482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.451460 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerStarted","Data":"9b31f4f6e027bb88572e799de4940b428b31200c5b8639f53e5502c7e0724db2"} Oct 29 23:00:03.453106 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.453086 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerStarted","Data":"44cbeac46b8feb33cab77e702451d56693b725c1e8567ce9c4f9d8f55192c70f"} Oct 29 23:00:03.458348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.458321 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerStarted","Data":"e3d8332341f0b5076edbd14c4bf50aca6ce3b0101524dea6e849695189af9776"} Oct 29 23:00:03.463921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.463903 5367 generic.go:356] "Generic (PLEG): container finished" podID="af08130d-e1d7-41eb-abe7-f626b698231f" containerID="740d11ee2f32c4d0a62e175f36ae6cfcedda7a30cc8983ee7dd668e7815e16d3" exitCode=60 Oct 29 23:00:03.464167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.464066 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerDied","Data":"740d11ee2f32c4d0a62e175f36ae6cfcedda7a30cc8983ee7dd668e7815e16d3"} Oct 29 23:00:03.464384 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.464366 5367 scope.go:122] "RemoveContainer" containerID="740d11ee2f32c4d0a62e175f36ae6cfcedda7a30cc8983ee7dd668e7815e16d3" Oct 29 23:00:03.468823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.468799 5367 generic.go:356] "Generic (PLEG): container finished" podID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerID="5eb02b7d190f217988258f79e053d5a0671323bd37587ec3bb96ac0c0329b482" exitCode=60 Oct 29 23:00:03.469036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.469018 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerDied","Data":"5eb02b7d190f217988258f79e053d5a0671323bd37587ec3bb96ac0c0329b482"} Oct 29 23:00:03.469155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.469083 5367 scope.go:122] "RemoveContainer" containerID="5eb02b7d190f217988258f79e053d5a0671323bd37587ec3bb96ac0c0329b482" Oct 29 23:00:03.472616 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.472595 5367 generic.go:356] "Generic (PLEG): container finished" podID="fbdec272-136f-4d21-a496-fe04aede5621" containerID="b7f6be789e988ff899331700196401c5b288e51542ac7c96779644592a9eba20" exitCode=60 Oct 29 23:00:03.472716 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.472669 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerDied","Data":"b7f6be789e988ff899331700196401c5b288e51542ac7c96779644592a9eba20"} Oct 29 23:00:03.472982 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.472963 5367 scope.go:122] "RemoveContainer" containerID="b7f6be789e988ff899331700196401c5b288e51542ac7c96779644592a9eba20" Oct 29 23:00:03.478952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.478911 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" podStartSLOduration=3.47889881 podStartE2EDuration="3.47889881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.475966891 +0000 UTC m=+2670.370041808" watchObservedRunningTime="2026-10-29 23:00:03.47889881 +0000 UTC m=+2670.372973730" Oct 29 23:00:03.523024 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.522981 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" podStartSLOduration=3.5229672990000003 podStartE2EDuration="3.522967299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.519378791 +0000 UTC m=+2670.413453713" watchObservedRunningTime="2026-10-29 23:00:03.522967299 +0000 UTC m=+2670.417042220" Oct 29 23:00:03.607367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.607320 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" podStartSLOduration=3.607305647 podStartE2EDuration="3.607305647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.603831493 +0000 UTC m=+2670.497906416" watchObservedRunningTime="2026-10-29 23:00:03.607305647 +0000 UTC m=+2670.501380562" Oct 29 23:00:03.686029 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.685930 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" podStartSLOduration=3.685914856 podStartE2EDuration="3.685914856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.684644688 +0000 UTC m=+2670.578719607" watchObservedRunningTime="2026-10-29 23:00:03.685914856 +0000 UTC m=+2670.579989775" Oct 29 23:00:03.777686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.777628 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" podStartSLOduration=3.777611675 podStartE2EDuration="3.777611675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.77419093 +0000 UTC m=+2670.668265849" watchObservedRunningTime="2026-10-29 23:00:03.777611675 +0000 UTC m=+2670.671686596" Oct 29 23:00:03.820204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.820159 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" podStartSLOduration=3.820145008 podStartE2EDuration="3.820145008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.816329295 +0000 UTC m=+2670.710404238" watchObservedRunningTime="2026-10-29 23:00:03.820145008 +0000 UTC m=+2670.714219927" Oct 29 23:00:03.919192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:03.919145 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" podStartSLOduration=3.91913298 podStartE2EDuration="3.91913298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.916247561 +0000 UTC m=+2670.810322482" watchObservedRunningTime="2026-10-29 23:00:03.91913298 +0000 UTC m=+2670.813207900" Oct 29 23:00:04.059083 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.059010 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" podStartSLOduration=4.058991208 podStartE2EDuration="4.058991208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.052260103 +0000 UTC m=+2670.946335026" watchObservedRunningTime="2026-10-29 23:00:04.058991208 +0000 UTC m=+2670.953066127" Oct 29 23:00:04.176166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.176074 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" podStartSLOduration=4.176056279 podStartE2EDuration="4.176056279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.1755245 +0000 UTC m=+2671.069599435" watchObservedRunningTime="2026-10-29 23:00:04.176056279 +0000 UTC m=+2671.070131201" Oct 29 23:00:04.485910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.485843 5367 generic.go:356] "Generic (PLEG): container finished" podID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerID="c944b7832c9cd2ea0298ed60ad3227aae982c4b56a19606ce54e72d56c30e37d" exitCode=60 Oct 29 23:00:04.486285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.485927 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerDied","Data":"c944b7832c9cd2ea0298ed60ad3227aae982c4b56a19606ce54e72d56c30e37d"} Oct 29 23:00:04.486349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.486313 5367 scope.go:122] "RemoveContainer" containerID="c944b7832c9cd2ea0298ed60ad3227aae982c4b56a19606ce54e72d56c30e37d" Oct 29 23:00:04.488547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.488490 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerStarted","Data":"7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df"} Oct 29 23:00:04.491688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.491662 5367 generic.go:356] "Generic (PLEG): container finished" podID="fe3ae329-788f-4719-b10e-146467086b36" containerID="fb2d1fa08ca4e7ec6e0d6bc7a81c6439dd5152915785081477ec2f9ad289a19e" exitCode=60 Oct 29 23:00:04.491825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.491754 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerDied","Data":"fb2d1fa08ca4e7ec6e0d6bc7a81c6439dd5152915785081477ec2f9ad289a19e"} Oct 29 23:00:04.492102 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.492085 5367 scope.go:122] "RemoveContainer" containerID="fb2d1fa08ca4e7ec6e0d6bc7a81c6439dd5152915785081477ec2f9ad289a19e" Oct 29 23:00:04.493491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.493472 5367 generic.go:356] "Generic (PLEG): container finished" podID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerID="ae8225938def214593173ce6e7f48328cef759dc8c6f2cc720af47012ae08838" exitCode=60 Oct 29 23:00:04.493595 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.493540 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerDied","Data":"ae8225938def214593173ce6e7f48328cef759dc8c6f2cc720af47012ae08838"} Oct 29 23:00:04.493856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.493838 5367 scope.go:122] "RemoveContainer" containerID="ae8225938def214593173ce6e7f48328cef759dc8c6f2cc720af47012ae08838" Oct 29 23:00:04.497707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.497673 5367 generic.go:356] "Generic (PLEG): container finished" podID="d119d843-8ea0-444b-9590-614da9a4f435" containerID="c7e447c24777ee271e5f4ae559fffe469b49a6c6ad0e6982768b955aed3ef9aa" exitCode=60 Oct 29 23:00:04.497797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.497782 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerDied","Data":"c7e447c24777ee271e5f4ae559fffe469b49a6c6ad0e6982768b955aed3ef9aa"} Oct 29 23:00:04.498114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.498095 5367 scope.go:122] "RemoveContainer" containerID="c7e447c24777ee271e5f4ae559fffe469b49a6c6ad0e6982768b955aed3ef9aa" Oct 29 23:00:04.500327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.500291 5367 generic.go:356] "Generic (PLEG): container finished" podID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerID="32f94865974c413d10f8605e90ff9f9d97b1e7b6af01faaf93186380ce719a8b" exitCode=60 Oct 29 23:00:04.500427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.500408 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerDied","Data":"32f94865974c413d10f8605e90ff9f9d97b1e7b6af01faaf93186380ce719a8b"} Oct 29 23:00:04.500748 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.500731 5367 scope.go:122] "RemoveContainer" containerID="32f94865974c413d10f8605e90ff9f9d97b1e7b6af01faaf93186380ce719a8b" Oct 29 23:00:04.509673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.509235 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerStarted","Data":"fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8"} Oct 29 23:00:04.514134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.514115 5367 generic.go:356] "Generic (PLEG): container finished" podID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerID="9b31f4f6e027bb88572e799de4940b428b31200c5b8639f53e5502c7e0724db2" exitCode=60 Oct 29 23:00:04.514214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.514166 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerDied","Data":"9b31f4f6e027bb88572e799de4940b428b31200c5b8639f53e5502c7e0724db2"} Oct 29 23:00:04.514420 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.514400 5367 scope.go:122] "RemoveContainer" containerID="9b31f4f6e027bb88572e799de4940b428b31200c5b8639f53e5502c7e0724db2" Oct 29 23:00:04.516772 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.516742 5367 generic.go:356] "Generic (PLEG): container finished" podID="81935cce-192e-4569-afe7-b61fe0a9a419" containerID="44cbeac46b8feb33cab77e702451d56693b725c1e8567ce9c4f9d8f55192c70f" exitCode=60 Oct 29 23:00:04.516865 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.516823 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerDied","Data":"44cbeac46b8feb33cab77e702451d56693b725c1e8567ce9c4f9d8f55192c70f"} Oct 29 23:00:04.517412 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.517058 5367 scope.go:122] "RemoveContainer" containerID="44cbeac46b8feb33cab77e702451d56693b725c1e8567ce9c4f9d8f55192c70f" Oct 29 23:00:04.522303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.521756 5367 generic.go:356] "Generic (PLEG): container finished" podID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerID="e3d8332341f0b5076edbd14c4bf50aca6ce3b0101524dea6e849695189af9776" exitCode=60 Oct 29 23:00:04.522303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.521869 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerDied","Data":"e3d8332341f0b5076edbd14c4bf50aca6ce3b0101524dea6e849695189af9776"} Oct 29 23:00:04.522303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.522207 5367 scope.go:122] "RemoveContainer" containerID="e3d8332341f0b5076edbd14c4bf50aca6ce3b0101524dea6e849695189af9776" Oct 29 23:00:04.538825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.538771 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerStarted","Data":"49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee"} Oct 29 23:00:04.542648 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.542624 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerStarted","Data":"49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3"} Oct 29 23:00:04.546394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.546368 5367 generic.go:356] "Generic (PLEG): container finished" podID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerID="a5886dbe1009ba84c712627ab6d8966400bc20bae7ff5337bd17b931148195ed" exitCode=60 Oct 29 23:00:04.546524 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.546453 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerDied","Data":"a5886dbe1009ba84c712627ab6d8966400bc20bae7ff5337bd17b931148195ed"} Oct 29 23:00:04.547117 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.546819 5367 scope.go:122] "RemoveContainer" containerID="a5886dbe1009ba84c712627ab6d8966400bc20bae7ff5337bd17b931148195ed" Oct 29 23:00:04.573299 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.573244 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerStarted","Data":"f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343"} Oct 29 23:00:04.579365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.578597 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerStarted","Data":"e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423"} Oct 29 23:00:04.670710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.670674 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl"] Oct 29 23:00:04.678804 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.678769 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.685725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.685692 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl"] Oct 29 23:00:04.696162 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.696133 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:04.759554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.757221 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" podStartSLOduration=4.757204006 podStartE2EDuration="4.757204006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.755757038 +0000 UTC m=+2671.649831958" watchObservedRunningTime="2026-10-29 23:00:04.757204006 +0000 UTC m=+2671.651278929" Oct 29 23:00:04.838647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.838612 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.838833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.838814 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.838925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.838911 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl92z\" (UniqueName: \"kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.940870 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.940840 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.941048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.941032 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.941152 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.941137 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl92z\" (UniqueName: \"kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.941871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.941848 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:04.943310 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:04.943291 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:05.056843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.055287 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl92z\" (UniqueName: \"kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z\") pod \"maas-api-revoke-sub-68befb42-sqfwl\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:05.198470 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.195451 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" podStartSLOduration=5.19543182 podStartE2EDuration="5.19543182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:05.184855956 +0000 UTC m=+2672.078930877" watchObservedRunningTime="2026-10-29 23:00:05.19543182 +0000 UTC m=+2672.089506742" Oct 29 23:00:05.342364 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.342305 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:05.599357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.595452 5367 generic.go:356] "Generic (PLEG): container finished" podID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerID="426414ca09baecf1e7e9f0c78c9649f07be5296a7c24aa12d2293ed2b6d8c534" exitCode=60 Oct 29 23:00:05.599357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.595559 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerDied","Data":"426414ca09baecf1e7e9f0c78c9649f07be5296a7c24aa12d2293ed2b6d8c534"} Oct 29 23:00:05.599357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.595606 5367 scope.go:122] "RemoveContainer" containerID="5eb02b7d190f217988258f79e053d5a0671323bd37587ec3bb96ac0c0329b482" Oct 29 23:00:05.599357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.596155 5367 scope.go:122] "RemoveContainer" containerID="426414ca09baecf1e7e9f0c78c9649f07be5296a7c24aa12d2293ed2b6d8c534" Oct 29 23:00:05.599357 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.596511 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95_odh-ai-gateway-infra(59a80962-cbc0-4103-a7ef-387f88af4fdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" Oct 29 23:00:05.601930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.601896 5367 generic.go:356] "Generic (PLEG): container finished" podID="fbdec272-136f-4d21-a496-fe04aede5621" containerID="49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3" exitCode=60 Oct 29 23:00:05.602049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.602024 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerDied","Data":"49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3"} Oct 29 23:00:05.602422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.602402 5367 scope.go:122] "RemoveContainer" containerID="49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3" Oct 29 23:00:05.602857 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.602835 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq_odh-ai-gateway-infra(fbdec272-136f-4d21-a496-fe04aede5621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" podUID="fbdec272-136f-4d21-a496-fe04aede5621" Oct 29 23:00:05.605744 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.605279 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerStarted","Data":"fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b"} Oct 29 23:00:05.609197 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.609174 5367 generic.go:356] "Generic (PLEG): container finished" podID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerID="f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343" exitCode=60 Oct 29 23:00:05.609445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.609278 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerDied","Data":"f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343"} Oct 29 23:00:05.609599 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.609582 5367 scope.go:122] "RemoveContainer" containerID="f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343" Oct 29 23:00:05.609936 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.609916 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn_odh-ai-gateway-infra(f64ffbae-aa0d-4478-8a93-07c0094d7c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" Oct 29 23:00:05.615198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.614016 5367 generic.go:356] "Generic (PLEG): container finished" podID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerID="e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423" exitCode=60 Oct 29 23:00:05.615198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.614091 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerDied","Data":"e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423"} Oct 29 23:00:05.615198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.614453 5367 scope.go:122] "RemoveContainer" containerID="e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423" Oct 29 23:00:05.615198 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.614838 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm_odh-ai-gateway-infra(3f056935-3821-4d4a-8fe6-7a9e41a9c6eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" Oct 29 23:00:05.619174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.618146 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerStarted","Data":"1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3"} Oct 29 23:00:05.627487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.625490 5367 scope.go:122] "RemoveContainer" containerID="b7f6be789e988ff899331700196401c5b288e51542ac7c96779644592a9eba20" Oct 29 23:00:05.641199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.636585 5367 generic.go:356] "Generic (PLEG): container finished" podID="722b2177-f08d-4203-b365-310ef1c795a5" containerID="7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df" exitCode=60 Oct 29 23:00:05.641199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.636673 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerDied","Data":"7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df"} Oct 29 23:00:05.641199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.637459 5367 scope.go:122] "RemoveContainer" containerID="7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df" Oct 29 23:00:05.641199 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.638098 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp_odh-ai-gateway-infra(722b2177-f08d-4203-b365-310ef1c795a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" podUID="722b2177-f08d-4203-b365-310ef1c795a5" Oct 29 23:00:05.647038 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.643560 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerStarted","Data":"549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02"} Oct 29 23:00:05.648649 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.648623 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerStarted","Data":"9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898"} Oct 29 23:00:05.650847 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.650808 5367 generic.go:356] "Generic (PLEG): container finished" podID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerID="45d6015d1363998864228b6bffa580b90352aecc1866579bae64887f8212b833" exitCode=60 Oct 29 23:00:05.650897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.650868 5367 scope.go:122] "RemoveContainer" containerID="b3f742901d93046fae67e24900e88a74ae7316c6ff03d497553a6182da3cc0c4" Oct 29 23:00:05.650944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.650909 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerDied","Data":"45d6015d1363998864228b6bffa580b90352aecc1866579bae64887f8212b833"} Oct 29 23:00:05.651383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.651357 5367 scope.go:122] "RemoveContainer" containerID="45d6015d1363998864228b6bffa580b90352aecc1866579bae64887f8212b833" Oct 29 23:00:05.651679 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.651659 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm_odh-ai-gateway-infra(5ee465e5-6f19-4d52-9e85-00e40b15bd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" Oct 29 23:00:05.662581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.662554 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl"] Oct 29 23:00:05.671647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.671008 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerStarted","Data":"72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364"} Oct 29 23:00:05.675781 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.675758 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerStarted","Data":"3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a"} Oct 29 23:00:05.678267 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.678243 5367 generic.go:356] "Generic (PLEG): container finished" podID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerID="fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8" exitCode=60 Oct 29 23:00:05.678358 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.678306 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerDied","Data":"fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8"} Oct 29 23:00:05.678622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.678606 5367 scope.go:122] "RemoveContainer" containerID="fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8" Oct 29 23:00:05.678873 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.678855 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m_odh-ai-gateway-infra(8003be9d-b7c7-4b0a-b988-02d13f22102b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" Oct 29 23:00:05.681148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.681123 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerStarted","Data":"02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4"} Oct 29 23:00:05.684287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.683726 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerStarted","Data":"91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1"} Oct 29 23:00:05.687527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.687202 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerStarted","Data":"cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d"} Oct 29 23:00:05.691131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.690398 5367 generic.go:356] "Generic (PLEG): container finished" podID="af08130d-e1d7-41eb-abe7-f626b698231f" containerID="49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee" exitCode=60 Oct 29 23:00:05.691131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.690442 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerDied","Data":"49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee"} Oct 29 23:00:05.691131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:05.690800 5367 scope.go:122] "RemoveContainer" containerID="49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee" Oct 29 23:00:05.691131 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:05.691070 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n_odh-ai-gateway-infra(af08130d-e1d7-41eb-abe7-f626b698231f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" Oct 29 23:00:06.036493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.036431 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb"] Oct 29 23:00:06.053016 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.052989 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4097d25f-tlpdb"] Oct 29 23:00:06.218196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.217975 5367 scope.go:122] "RemoveContainer" containerID="08cc882e1eaae1dae9a5fb683b879bf690c655267421b1ac9712a5b046c6d5a3" Oct 29 23:00:06.236577 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.236549 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 23:00:06.236991 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.236975 5367 scope.go:122] "RemoveContainer" containerID="b6832e89b46d0df3f18703c64dc7b0e87f05326bceef823c0d8b70ad703eb2a2" Oct 29 23:00:06.252282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.252259 5367 scope.go:122] "RemoveContainer" containerID="acf9deb4731c25438853f977f018d9f9aea80c2a52aaac96b0f9d1c1f0bfbcc7" Oct 29 23:00:06.266796 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.266778 5367 scope.go:122] "RemoveContainer" containerID="f461b696336a5add6695675a55b730168f4fce62ee091dde0da0dfdd7b6900f0" Oct 29 23:00:06.280130 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.280110 5367 scope.go:122] "RemoveContainer" containerID="740d11ee2f32c4d0a62e175f36ae6cfcedda7a30cc8983ee7dd668e7815e16d3" Oct 29 23:00:06.332528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.325173 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg\" (UniqueName: \"kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg\") pod \"e039e8c1-7cae-4480-97bd-4981a3425750\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " Oct 29 23:00:06.332528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.325523 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume\") pod \"e039e8c1-7cae-4480-97bd-4981a3425750\" (UID: \"e039e8c1-7cae-4480-97bd-4981a3425750\") " Oct 29 23:00:06.332528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.332047 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume" pod "e039e8c1-7cae-4480-97bd-4981a3425750" (UID: "e039e8c1-7cae-4480-97bd-4981a3425750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:06.339556 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.339510 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg" pod "e039e8c1-7cae-4480-97bd-4981a3425750" (UID: "e039e8c1-7cae-4480-97bd-4981a3425750"). InnerVolumeSpecName "kube-api-access-xljwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:06.429293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.429262 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljwg\" (UniqueName: \"kubernetes.io/projected/e039e8c1-7cae-4480-97bd-4981a3425750-kube-api-access-xljwg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:06.429293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.429290 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e039e8c1-7cae-4480-97bd-4981a3425750-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:06.776798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.766118 5367 generic.go:356] "Generic (PLEG): container finished" podID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerID="1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3" exitCode=60 Oct 29 23:00:06.776798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.766223 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerDied","Data":"1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3"} Oct 29 23:00:06.776798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.766278 5367 scope.go:122] "RemoveContainer" containerID="c944b7832c9cd2ea0298ed60ad3227aae982c4b56a19606ce54e72d56c30e37d" Oct 29 23:00:06.776798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.766921 5367 scope.go:122] "RemoveContainer" containerID="1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3" Oct 29 23:00:06.776798 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.767426 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk_odh-ai-gateway-infra(ac6c78c9-5de2-4bc5-91e5-9f891eb598b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" Oct 29 23:00:06.804745 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.802210 5367 scope.go:122] "RemoveContainer" containerID="7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df" Oct 29 23:00:06.804745 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.803017 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp_odh-ai-gateway-infra(722b2177-f08d-4203-b365-310ef1c795a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" podUID="722b2177-f08d-4203-b365-310ef1c795a5" Oct 29 23:00:06.824772 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.824726 5367 generic.go:356] "Generic (PLEG): container finished" podID="fe3ae329-788f-4719-b10e-146467086b36" containerID="549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02" exitCode=60 Oct 29 23:00:06.824930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.824818 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerDied","Data":"549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02"} Oct 29 23:00:06.825158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.825142 5367 scope.go:122] "RemoveContainer" containerID="549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02" Oct 29 23:00:06.825633 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.825484 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56_odh-ai-gateway-infra(fe3ae329-788f-4719-b10e-146467086b36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" podUID="fe3ae329-788f-4719-b10e-146467086b36" Oct 29 23:00:06.827851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.827821 5367 generic.go:356] "Generic (PLEG): container finished" podID="e039e8c1-7cae-4480-97bd-4981a3425750" containerID="56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7" exitCode=0 Oct 29 23:00:06.827960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.827923 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" event={"ID":"e039e8c1-7cae-4480-97bd-4981a3425750","Type":"ContainerDied","Data":"56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7"} Oct 29 23:00:06.828017 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.827961 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" event={"ID":"e039e8c1-7cae-4480-97bd-4981a3425750","Type":"ContainerDied","Data":"a3e407541a68fbcf5705b6c61d4708defb0cacd06ae8dbe4ccec9bcc1ef23a4d"} Oct 29 23:00:06.828068 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.828037 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp" Oct 29 23:00:06.831206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.831183 5367 generic.go:356] "Generic (PLEG): container finished" podID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerID="9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898" exitCode=60 Oct 29 23:00:06.831393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.831375 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerDied","Data":"9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898"} Oct 29 23:00:06.833388 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.831792 5367 scope.go:122] "RemoveContainer" containerID="9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898" Oct 29 23:00:06.833916 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.833894 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp_odh-ai-gateway-infra(7531bdb4-381a-4039-bcea-5ed7379a8546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" Oct 29 23:00:06.835377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.835358 5367 scope.go:122] "RemoveContainer" containerID="fb2d1fa08ca4e7ec6e0d6bc7a81c6439dd5152915785081477ec2f9ad289a19e" Oct 29 23:00:06.838193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.838170 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" event={"ID":"46a5c283-414a-4b5f-84b2-fd0c7f071617","Type":"ContainerStarted","Data":"35d55d84e75f10de7292bfcfc1cf6ec46446120cc0b7ceacf2c70f2c00c0dc20"} Oct 29 23:00:06.838346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.838328 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" event={"ID":"46a5c283-414a-4b5f-84b2-fd0c7f071617","Type":"ContainerStarted","Data":"8a09b5eed25950cdac4dcec1ac3aa4e01a3a51ec77aaa15cff6a822cb95af067"} Oct 29 23:00:06.841283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.840693 5367 generic.go:356] "Generic (PLEG): container finished" podID="d119d843-8ea0-444b-9590-614da9a4f435" containerID="72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364" exitCode=60 Oct 29 23:00:06.841283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.840793 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerDied","Data":"72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364"} Oct 29 23:00:06.841717 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.841508 5367 scope.go:122] "RemoveContainer" containerID="72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364" Oct 29 23:00:06.842444 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.841879 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7_odh-ai-gateway-infra(d119d843-8ea0-444b-9590-614da9a4f435)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" podUID="d119d843-8ea0-444b-9590-614da9a4f435" Oct 29 23:00:06.848993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.848968 5367 generic.go:356] "Generic (PLEG): container finished" podID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerID="3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a" exitCode=60 Oct 29 23:00:06.849462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.849029 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerDied","Data":"3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a"} Oct 29 23:00:06.849547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.849283 5367 scope.go:122] "RemoveContainer" containerID="3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a" Oct 29 23:00:06.849849 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.849828 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5_odh-ai-gateway-infra(9ba7d901-6282-41be-ad0f-fd74055bad81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" Oct 29 23:00:06.851617 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.851602 5367 scope.go:122] "RemoveContainer" containerID="fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8" Oct 29 23:00:06.851912 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.851890 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m_odh-ai-gateway-infra(8003be9d-b7c7-4b0a-b988-02d13f22102b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" Oct 29 23:00:06.853287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.853269 5367 generic.go:356] "Generic (PLEG): container finished" podID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerID="02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4" exitCode=60 Oct 29 23:00:06.853383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.853333 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerDied","Data":"02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4"} Oct 29 23:00:06.853508 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.853478 5367 scope.go:122] "RemoveContainer" containerID="02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4" Oct 29 23:00:06.853739 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.853686 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5_odh-ai-gateway-infra(9cdf2542-f96c-4c37-88e7-fdb0a7778b43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" Oct 29 23:00:06.854171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.854150 5367 scope.go:122] "RemoveContainer" containerID="56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7" Oct 29 23:00:06.863257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.859813 5367 generic.go:356] "Generic (PLEG): container finished" podID="81935cce-192e-4569-afe7-b61fe0a9a419" containerID="91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1" exitCode=60 Oct 29 23:00:06.863257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.859908 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerDied","Data":"91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1"} Oct 29 23:00:06.863257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.860097 5367 scope.go:122] "RemoveContainer" containerID="91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1" Oct 29 23:00:06.863257 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.860343 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb_odh-ai-gateway-infra(81935cce-192e-4569-afe7-b61fe0a9a419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" Oct 29 23:00:06.865052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.865031 5367 generic.go:356] "Generic (PLEG): container finished" podID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerID="cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d" exitCode=60 Oct 29 23:00:06.865170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.865128 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerDied","Data":"cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d"} Oct 29 23:00:06.865348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.865330 5367 scope.go:122] "RemoveContainer" containerID="cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d" Oct 29 23:00:06.865607 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.865588 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm_odh-ai-gateway-infra(76ba151d-e166-4c58-8d4e-6e507a0986e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" Oct 29 23:00:06.871360 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.871331 5367 scope.go:122] "RemoveContainer" containerID="56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7" Oct 29 23:00:06.871666 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.871631 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7\": container with ID starting with 56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7 not found: ID does not exist" containerID="56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7" Oct 29 23:00:06.871783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.871666 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7"} err="failed to get container status \"56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7\": rpc error: code = NotFound desc = could not find container \"56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7\": container with ID starting with 56f64a517f0a52bf5552956d039235ff7739d9cc18ac48030da94943ef8de3e7 not found: ID does not exist" Oct 29 23:00:06.871783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.871686 5367 scope.go:122] "RemoveContainer" containerID="ae8225938def214593173ce6e7f48328cef759dc8c6f2cc720af47012ae08838" Oct 29 23:00:06.873093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.873074 5367 scope.go:122] "RemoveContainer" containerID="49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee" Oct 29 23:00:06.873341 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.873318 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n_odh-ai-gateway-infra(af08130d-e1d7-41eb-abe7-f626b698231f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" Oct 29 23:00:06.875125 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.875106 5367 scope.go:122] "RemoveContainer" containerID="426414ca09baecf1e7e9f0c78c9649f07be5296a7c24aa12d2293ed2b6d8c534" Oct 29 23:00:06.875406 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.875351 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95_odh-ai-gateway-infra(59a80962-cbc0-4103-a7ef-387f88af4fdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" Oct 29 23:00:06.876903 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.876875 5367 scope.go:122] "RemoveContainer" containerID="49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3" Oct 29 23:00:06.877115 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.877097 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq_odh-ai-gateway-infra(fbdec272-136f-4d21-a496-fe04aede5621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" podUID="fbdec272-136f-4d21-a496-fe04aede5621" Oct 29 23:00:06.878656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.878633 5367 generic.go:356] "Generic (PLEG): container finished" podID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerID="fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b" exitCode=60 Oct 29 23:00:06.878759 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.878737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerDied","Data":"fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b"} Oct 29 23:00:06.879019 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.878994 5367 scope.go:122] "RemoveContainer" containerID="fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b" Oct 29 23:00:06.879288 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.879262 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj_odh-ai-gateway-infra(def8b4aa-4cbf-43d5-ac35-43a87301a566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" Oct 29 23:00:06.880829 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.880813 5367 scope.go:122] "RemoveContainer" containerID="f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343" Oct 29 23:00:06.881049 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.881027 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn_odh-ai-gateway-infra(f64ffbae-aa0d-4478-8a93-07c0094d7c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" Oct 29 23:00:06.882710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.882680 5367 scope.go:122] "RemoveContainer" containerID="e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423" Oct 29 23:00:06.883012 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:06.882990 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm_odh-ai-gateway-infra(3f056935-3821-4d4a-8fe6-7a9e41a9c6eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" Oct 29 23:00:06.888070 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.888050 5367 scope.go:122] "RemoveContainer" containerID="c7e447c24777ee271e5f4ae559fffe469b49a6c6ad0e6982768b955aed3ef9aa" Oct 29 23:00:06.902077 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.902057 5367 scope.go:122] "RemoveContainer" containerID="32f94865974c413d10f8605e90ff9f9d97b1e7b6af01faaf93186380ce719a8b" Oct 29 23:00:06.916072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.916049 5367 scope.go:122] "RemoveContainer" containerID="9b31f4f6e027bb88572e799de4940b428b31200c5b8639f53e5502c7e0724db2" Oct 29 23:00:06.930794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.930777 5367 scope.go:122] "RemoveContainer" containerID="44cbeac46b8feb33cab77e702451d56693b725c1e8567ce9c4f9d8f55192c70f" Oct 29 23:00:06.945136 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.945116 5367 scope.go:122] "RemoveContainer" containerID="e3d8332341f0b5076edbd14c4bf50aca6ce3b0101524dea6e849695189af9776" Oct 29 23:00:06.961732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:06.961716 5367 scope.go:122] "RemoveContainer" containerID="a5886dbe1009ba84c712627ab6d8966400bc20bae7ff5337bd17b931148195ed" Oct 29 23:00:07.228451 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.228418 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24487aa1-d378-419e-ba93-2b5b28013699" path="/var/lib/kubelet/pods/24487aa1-d378-419e-ba93-2b5b28013699/volumes" Oct 29 23:00:07.351530 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.351494 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp"] Oct 29 23:00:07.378563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.378527 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-40cf9a-5ccdc94fff-2wxnp"] Oct 29 23:00:07.644124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.644061 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" podStartSLOduration=3.644042992 podStartE2EDuration="3.644042992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:07.6383796 +0000 UTC m=+2674.532454524" watchObservedRunningTime="2026-10-29 23:00:07.644042992 +0000 UTC m=+2674.538117912" Oct 29 23:00:07.893325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.893287 5367 scope.go:122] "RemoveContainer" containerID="fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b" Oct 29 23:00:07.893836 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.893801 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj_odh-ai-gateway-infra(def8b4aa-4cbf-43d5-ac35-43a87301a566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" Oct 29 23:00:07.895274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.895215 5367 scope.go:122] "RemoveContainer" containerID="1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3" Oct 29 23:00:07.895656 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.895627 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk_odh-ai-gateway-infra(ac6c78c9-5de2-4bc5-91e5-9f891eb598b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" Oct 29 23:00:07.897601 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.897572 5367 scope.go:122] "RemoveContainer" containerID="549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02" Oct 29 23:00:07.898399 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.898370 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56_odh-ai-gateway-infra(fe3ae329-788f-4719-b10e-146467086b36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" podUID="fe3ae329-788f-4719-b10e-146467086b36" Oct 29 23:00:07.924745 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.924717 5367 scope.go:122] "RemoveContainer" containerID="9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898" Oct 29 23:00:07.925710 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.925654 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp_odh-ai-gateway-infra(7531bdb4-381a-4039-bcea-5ed7379a8546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" Oct 29 23:00:07.928426 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.928399 5367 generic.go:356] "Generic (PLEG): container finished" podID="46a5c283-414a-4b5f-84b2-fd0c7f071617" containerID="35d55d84e75f10de7292bfcfc1cf6ec46446120cc0b7ceacf2c70f2c00c0dc20" exitCode=0 Oct 29 23:00:07.928723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.928685 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" event={"ID":"46a5c283-414a-4b5f-84b2-fd0c7f071617","Type":"ContainerDied","Data":"35d55d84e75f10de7292bfcfc1cf6ec46446120cc0b7ceacf2c70f2c00c0dc20"} Oct 29 23:00:07.967977 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.967896 5367 scope.go:122] "RemoveContainer" containerID="72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364" Oct 29 23:00:07.968300 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.968276 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7_odh-ai-gateway-infra(d119d843-8ea0-444b-9590-614da9a4f435)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" podUID="d119d843-8ea0-444b-9590-614da9a4f435" Oct 29 23:00:07.970156 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.970135 5367 scope.go:122] "RemoveContainer" containerID="3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a" Oct 29 23:00:07.970419 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.970391 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5_odh-ai-gateway-infra(9ba7d901-6282-41be-ad0f-fd74055bad81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" Oct 29 23:00:07.972863 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.972843 5367 scope.go:122] "RemoveContainer" containerID="02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4" Oct 29 23:00:07.973112 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.973085 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5_odh-ai-gateway-infra(9cdf2542-f96c-4c37-88e7-fdb0a7778b43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" Oct 29 23:00:07.975609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.975591 5367 scope.go:122] "RemoveContainer" containerID="91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1" Oct 29 23:00:07.975914 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.975889 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb_odh-ai-gateway-infra(81935cce-192e-4569-afe7-b61fe0a9a419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" Oct 29 23:00:07.977248 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:07.977232 5367 scope.go:122] "RemoveContainer" containerID="cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d" Oct 29 23:00:07.977431 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:07.977413 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm_odh-ai-gateway-infra(76ba151d-e166-4c58-8d4e-6e507a0986e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" Oct 29 23:00:09.075738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.072790 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q"] Oct 29 23:00:09.114202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.111098 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e49a879-hjj7q"] Oct 29 23:00:09.228505 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.228478 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:09.230496 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.230457 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0827f96b-317f-4abc-b268-3ded5f8cea8d" path="/var/lib/kubelet/pods/0827f96b-317f-4abc-b268-3ded5f8cea8d/volumes" Oct 29 23:00:09.231288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.231261 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" path="/var/lib/kubelet/pods/e039e8c1-7cae-4480-97bd-4981a3425750/volumes" Oct 29 23:00:09.340820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.340787 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z\" (UniqueName: \"kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z\") pod \"46a5c283-414a-4b5f-84b2-fd0c7f071617\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " Oct 29 23:00:09.340990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.340974 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca\") pod \"46a5c283-414a-4b5f-84b2-fd0c7f071617\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " Oct 29 23:00:09.341060 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.341044 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca\") pod \"46a5c283-414a-4b5f-84b2-fd0c7f071617\" (UID: \"46a5c283-414a-4b5f-84b2-fd0c7f071617\") " Oct 29 23:00:09.341404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.341372 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca" pod "46a5c283-414a-4b5f-84b2-fd0c7f071617" (UID: "46a5c283-414a-4b5f-84b2-fd0c7f071617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:09.341940 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.341912 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.343667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.343637 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca" pod "46a5c283-414a-4b5f-84b2-fd0c7f071617" (UID: "46a5c283-414a-4b5f-84b2-fd0c7f071617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:09.343793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.343743 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z" pod "46a5c283-414a-4b5f-84b2-fd0c7f071617" (UID: "46a5c283-414a-4b5f-84b2-fd0c7f071617"). InnerVolumeSpecName "kube-api-access-fl92z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:09.443951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.443915 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl92z\" (UniqueName: \"kubernetes.io/projected/46a5c283-414a-4b5f-84b2-fd0c7f071617-kube-api-access-fl92z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.443951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.443946 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a5c283-414a-4b5f-84b2-fd0c7f071617-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.996685 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.996650 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" Oct 29 23:00:09.996874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.996712 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl" event={"ID":"46a5c283-414a-4b5f-84b2-fd0c7f071617","Type":"ContainerDied","Data":"8a09b5eed25950cdac4dcec1ac3aa4e01a3a51ec77aaa15cff6a822cb95af067"} Oct 29 23:00:09.996874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:09.996765 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a09b5eed25950cdac4dcec1ac3aa4e01a3a51ec77aaa15cff6a822cb95af067" Oct 29 23:00:14.048661 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.048630 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5"] Oct 29 23:00:14.049529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.049511 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" containerName="payload-pre-processing" Oct 29 23:00:14.049590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.049531 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" containerName="payload-pre-processing" Oct 29 23:00:14.049590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.049575 5367 cpu_manager.go:414] "containerMap: removing container" podUID="46a5c283-414a-4b5f-84b2-fd0c7f071617" containerName="revoke-keys" Oct 29 23:00:14.049590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.049580 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a5c283-414a-4b5f-84b2-fd0c7f071617" containerName="revoke-keys" Oct 29 23:00:14.049734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.049722 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a5c283-414a-4b5f-84b2-fd0c7f071617" containerName="revoke-keys" Oct 29 23:00:14.049780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.049748 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e039e8c1-7cae-4480-97bd-4981a3425750" containerName="payload-pre-processing" Oct 29 23:00:14.054901 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.054884 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.072790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.072769 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8f1627-openshift-default-dockercfg-qjqk5\"" Oct 29 23:00:14.105742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.095487 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5"] Oct 29 23:00:14.113064 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113037 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjn6g\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113073 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113154 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113364 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113341 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113379 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113501 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113487 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113518 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113673 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.113809 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.113795 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.215197 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215162 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.215403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215251 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.215403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215309 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjn6g\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.215403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215342 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.215606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215406 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.215876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215836 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216026 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.215979 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216026 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216003 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216072 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216100 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216490 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216467 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216629 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216511 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216745 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.216839 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.216827 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.218000 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.217976 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.218121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.218104 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.235608 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.235586 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.235916 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.235895 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjn6g\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.369630 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.369602 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:14.687969 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:14.687938 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278233b5_18e7_44bf_940f_76c1e6ee8c42.slice/crio-b3920382562bb4a74fed48e5bb703b11acaa83fd3d16c6c1df6d0214c353d3c7 WatchSource:0}: Error finding container b3920382562bb4a74fed48e5bb703b11acaa83fd3d16c6c1df6d0214c353d3c7: Status 404 returned error can't find the container with id b3920382562bb4a74fed48e5bb703b11acaa83fd3d16c6c1df6d0214c353d3c7 Oct 29 23:00:14.689419 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.689389 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5"] Oct 29 23:00:14.690930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.690895 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:14.691028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.690963 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:14.691028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:14.690991 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:15.051331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.051237 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" event={"ID":"278233b5-18e7-44bf-940f-76c1e6ee8c42","Type":"ContainerStarted","Data":"316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74"} Oct 29 23:00:15.051331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.051306 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" event={"ID":"278233b5-18e7-44bf-940f-76c1e6ee8c42","Type":"ContainerStarted","Data":"b3920382562bb4a74fed48e5bb703b11acaa83fd3d16c6c1df6d0214c353d3c7"} Oct 29 23:00:15.119289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.119229 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podStartSLOduration=1.1191956730000001 podStartE2EDuration="1.119195673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:15.118253164 +0000 UTC m=+2682.012328094" watchObservedRunningTime="2026-10-29 23:00:15.119195673 +0000 UTC m=+2682.013270594" Oct 29 23:00:15.234558 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.234518 5367 scope.go:122] "RemoveContainer" containerID="7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df" Oct 29 23:00:15.235535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.235507 5367 scope.go:122] "RemoveContainer" containerID="fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8" Oct 29 23:00:15.384254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.384140 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:15.397202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.396836 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body= Oct 29 23:00:15.397202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:15.396902 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" Oct 29 23:00:16.090494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.090450 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerStarted","Data":"0aacbbac5ab399815674899d09c4bec0ce0b82c0b96335075b93dc41f4db24cc"} Oct 29 23:00:16.092755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.092728 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerStarted","Data":"94f625fca2be71ec9e2d442786045d065bf4367098af7402af82ad6f767cd5ae"} Oct 29 23:00:16.234722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.230485 5367 scope.go:122] "RemoveContainer" containerID="49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee" Oct 29 23:00:16.234722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.231984 5367 scope.go:122] "RemoveContainer" containerID="426414ca09baecf1e7e9f0c78c9649f07be5296a7c24aa12d2293ed2b6d8c534" Oct 29 23:00:16.234722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.234406 5367 scope.go:122] "RemoveContainer" containerID="cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d" Oct 29 23:00:16.385775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.369902 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body= Oct 29 23:00:16.385775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:16.370002 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" Oct 29 23:00:16.418293 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:16.418262 5367 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722b2177_f08d_4203_b365_310ef1c795a5.slice/crio-94f625fca2be71ec9e2d442786045d065bf4367098af7402af82ad6f767cd5ae.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:00:17.084884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.084835 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd"] Oct 29 23:00:17.105374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.105344 5367 generic.go:356] "Generic (PLEG): container finished" podID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerID="0aacbbac5ab399815674899d09c4bec0ce0b82c0b96335075b93dc41f4db24cc" exitCode=60 Oct 29 23:00:17.105831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.105419 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerDied","Data":"0aacbbac5ab399815674899d09c4bec0ce0b82c0b96335075b93dc41f4db24cc"} Oct 29 23:00:17.105831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.105486 5367 scope.go:122] "RemoveContainer" containerID="fb765551fb69057b20c0ef768d9039475b6a364811dc84749922b1b0f39f3fb8" Oct 29 23:00:17.105956 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.105940 5367 scope.go:122] "RemoveContainer" containerID="0aacbbac5ab399815674899d09c4bec0ce0b82c0b96335075b93dc41f4db24cc" Oct 29 23:00:17.106392 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:17.106365 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m_odh-ai-gateway-infra(8003be9d-b7c7-4b0a-b988-02d13f22102b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" Oct 29 23:00:17.107764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.107742 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerStarted","Data":"14629b1ce6105c3002cb34ec1a1e5e6201557e553fc69cc7cfe184846fd7ee09"} Oct 29 23:00:17.109736 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.109686 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerStarted","Data":"8c1267c19edd1ebb679d5d6413f0a6f4ef485ab942971e2f6c4cf6c7725a03f8"} Oct 29 23:00:17.111687 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.111666 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerStarted","Data":"c081b1cefb8f2d41c57da9873785dbf6e8b3ae04a58b41d1fcb30dd64336cfe1"} Oct 29 23:00:17.113598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.113579 5367 generic.go:356] "Generic (PLEG): container finished" podID="722b2177-f08d-4203-b365-310ef1c795a5" containerID="94f625fca2be71ec9e2d442786045d065bf4367098af7402af82ad6f767cd5ae" exitCode=60 Oct 29 23:00:17.113693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.113645 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerDied","Data":"94f625fca2be71ec9e2d442786045d065bf4367098af7402af82ad6f767cd5ae"} Oct 29 23:00:17.114086 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.114069 5367 scope.go:122] "RemoveContainer" containerID="94f625fca2be71ec9e2d442786045d065bf4367098af7402af82ad6f767cd5ae" Oct 29 23:00:17.114368 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:17.114343 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp_odh-ai-gateway-infra(722b2177-f08d-4203-b365-310ef1c795a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" podUID="722b2177-f08d-4203-b365-310ef1c795a5" Oct 29 23:00:17.122227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.122202 5367 scope.go:122] "RemoveContainer" containerID="7ae45ba40f5ee8ac13dd881810e45fa68dfb3c1662b14108bbd6bdf9eae1a9df" Oct 29 23:00:17.168723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.153274 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m"] Oct 29 23:00:17.226057 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.225458 5367 scope.go:122] "RemoveContainer" containerID="e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423" Oct 29 23:00:17.229856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.229658 5367 scope.go:122] "RemoveContainer" containerID="549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02" Oct 29 23:00:17.232392 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.230364 5367 scope.go:122] "RemoveContainer" containerID="49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3" Oct 29 23:00:17.232392 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.230634 5367 scope.go:122] "RemoveContainer" containerID="02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4" Oct 29 23:00:17.316720 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.315815 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp"] Oct 29 23:00:17.384727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.372737 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body= Oct 29 23:00:17.384727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:17.372790 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" Oct 29 23:00:18.126358 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.126327 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerStarted","Data":"373dda7c65045822d698d6b68ce7b1da9a012e99ae2fdcbaa9a46bd8cf1750d0"} Oct 29 23:00:18.128619 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.128595 5367 generic.go:356] "Generic (PLEG): container finished" podID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerID="14629b1ce6105c3002cb34ec1a1e5e6201557e553fc69cc7cfe184846fd7ee09" exitCode=60 Oct 29 23:00:18.128769 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.128660 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerDied","Data":"14629b1ce6105c3002cb34ec1a1e5e6201557e553fc69cc7cfe184846fd7ee09"} Oct 29 23:00:18.128769 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.128721 5367 scope.go:122] "RemoveContainer" containerID="cd8ac4dd1648a6f253e51bd77b6394b73f861497f8d7bce9cc3caafb5923aa0d" Oct 29 23:00:18.129114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.129093 5367 scope.go:122] "RemoveContainer" containerID="14629b1ce6105c3002cb34ec1a1e5e6201557e553fc69cc7cfe184846fd7ee09" Oct 29 23:00:18.129514 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:18.129483 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm_odh-ai-gateway-infra(76ba151d-e166-4c58-8d4e-6e507a0986e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" Oct 29 23:00:18.130967 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.130949 5367 generic.go:356] "Generic (PLEG): container finished" podID="af08130d-e1d7-41eb-abe7-f626b698231f" containerID="8c1267c19edd1ebb679d5d6413f0a6f4ef485ab942971e2f6c4cf6c7725a03f8" exitCode=60 Oct 29 23:00:18.131042 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.131021 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerDied","Data":"8c1267c19edd1ebb679d5d6413f0a6f4ef485ab942971e2f6c4cf6c7725a03f8"} Oct 29 23:00:18.131459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.131438 5367 scope.go:122] "RemoveContainer" containerID="8c1267c19edd1ebb679d5d6413f0a6f4ef485ab942971e2f6c4cf6c7725a03f8" Oct 29 23:00:18.131877 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:18.131856 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n_odh-ai-gateway-infra(af08130d-e1d7-41eb-abe7-f626b698231f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" Oct 29 23:00:18.133114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.133095 5367 generic.go:356] "Generic (PLEG): container finished" podID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerID="c081b1cefb8f2d41c57da9873785dbf6e8b3ae04a58b41d1fcb30dd64336cfe1" exitCode=60 Oct 29 23:00:18.133409 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.133161 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerDied","Data":"c081b1cefb8f2d41c57da9873785dbf6e8b3ae04a58b41d1fcb30dd64336cfe1"} Oct 29 23:00:18.133505 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.133489 5367 scope.go:122] "RemoveContainer" containerID="c081b1cefb8f2d41c57da9873785dbf6e8b3ae04a58b41d1fcb30dd64336cfe1" Oct 29 23:00:18.133873 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:18.133849 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95_odh-ai-gateway-infra(59a80962-cbc0-4103-a7ef-387f88af4fdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" Oct 29 23:00:18.135383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.135362 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerStarted","Data":"0c7840c42080716e9417c22c48f4f507596513fdc3a00fb5df3ab596a3644ef6"} Oct 29 23:00:18.137667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.137648 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerStarted","Data":"267bd81ad6ee2dd890d177c90c49b3594be39fef569b44b470417d385a85ccda"} Oct 29 23:00:18.142654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.142619 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerStarted","Data":"335b40b56383959ebea7641f5fccf9deff0c84a9c848ad1225073a113fb5cbe1"} Oct 29 23:00:18.145874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.145858 5367 scope.go:122] "RemoveContainer" containerID="49a433f3b1be4f8ade90f50d35b065671437b0e39389b93eb7beb4e2840a77ee" Oct 29 23:00:18.250777 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.234455 5367 scope.go:122] "RemoveContainer" containerID="45d6015d1363998864228b6bffa580b90352aecc1866579bae64887f8212b833" Oct 29 23:00:18.320273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.320096 5367 scope.go:122] "RemoveContainer" containerID="426414ca09baecf1e7e9f0c78c9649f07be5296a7c24aa12d2293ed2b6d8c534" Oct 29 23:00:18.333261 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.333234 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95"] Oct 29 23:00:18.396155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.395797 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body= Oct 29 23:00:18.396155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.396035 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" Oct 29 23:00:18.400452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.400426 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n"] Oct 29 23:00:18.475735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.461674 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm"] Oct 29 23:00:18.524375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.524304 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:18.607188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.607158 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca\") pod \"8003be9d-b7c7-4b0a-b988-02d13f22102b\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " Oct 29 23:00:18.607319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.607243 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr\" (UniqueName: \"kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr\") pod \"8003be9d-b7c7-4b0a-b988-02d13f22102b\" (UID: \"8003be9d-b7c7-4b0a-b988-02d13f22102b\") " Oct 29 23:00:18.608851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.608827 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca" pod "8003be9d-b7c7-4b0a-b988-02d13f22102b" (UID: "8003be9d-b7c7-4b0a-b988-02d13f22102b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.610115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.610091 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr" pod "8003be9d-b7c7-4b0a-b988-02d13f22102b" (UID: "8003be9d-b7c7-4b0a-b988-02d13f22102b"). InnerVolumeSpecName "kube-api-access-rxrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.684783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.683961 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:18.709046 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.709019 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca\") pod \"722b2177-f08d-4203-b365-310ef1c795a5\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " Oct 29 23:00:18.709207 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.709191 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm\" (UniqueName: \"kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm\") pod \"722b2177-f08d-4203-b365-310ef1c795a5\" (UID: \"722b2177-f08d-4203-b365-310ef1c795a5\") " Oct 29 23:00:18.709454 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.709423 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca" pod "722b2177-f08d-4203-b365-310ef1c795a5" (UID: "722b2177-f08d-4203-b365-310ef1c795a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.710080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.710052 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8003be9d-b7c7-4b0a-b988-02d13f22102b-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.710170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.710084 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrtr\" (UniqueName: \"kubernetes.io/projected/8003be9d-b7c7-4b0a-b988-02d13f22102b-kube-api-access-rxrtr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.710170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.710098 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722b2177-f08d-4203-b365-310ef1c795a5-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.711341 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.711318 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm" pod "722b2177-f08d-4203-b365-310ef1c795a5" (UID: "722b2177-f08d-4203-b365-310ef1c795a5"). InnerVolumeSpecName "kube-api-access-wk7vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.812019 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:18.811988 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7vm\" (UniqueName: \"kubernetes.io/projected/722b2177-f08d-4203-b365-310ef1c795a5-kube-api-access-wk7vm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.183786 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.183749 5367 generic.go:356] "Generic (PLEG): container finished" podID="fbdec272-136f-4d21-a496-fe04aede5621" containerID="0c7840c42080716e9417c22c48f4f507596513fdc3a00fb5df3ab596a3644ef6" exitCode=60 Oct 29 23:00:19.184249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.183930 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerDied","Data":"0c7840c42080716e9417c22c48f4f507596513fdc3a00fb5df3ab596a3644ef6"} Oct 29 23:00:19.184249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.184071 5367 scope.go:122] "RemoveContainer" containerID="49c9b8e4d129c8f61ca28f35d751ba593b50a16503fab8f2c17b6596779338e3" Oct 29 23:00:19.185014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.184990 5367 scope.go:122] "RemoveContainer" containerID="0c7840c42080716e9417c22c48f4f507596513fdc3a00fb5df3ab596a3644ef6" Oct 29 23:00:19.185649 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:19.185605 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq_odh-ai-gateway-infra(fbdec272-136f-4d21-a496-fe04aede5621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" podUID="fbdec272-136f-4d21-a496-fe04aede5621" Oct 29 23:00:19.188679 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.188657 5367 generic.go:356] "Generic (PLEG): container finished" podID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerID="267bd81ad6ee2dd890d177c90c49b3594be39fef569b44b470417d385a85ccda" exitCode=60 Oct 29 23:00:19.188822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.188803 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerDied","Data":"267bd81ad6ee2dd890d177c90c49b3594be39fef569b44b470417d385a85ccda"} Oct 29 23:00:19.189268 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.189252 5367 scope.go:122] "RemoveContainer" containerID="267bd81ad6ee2dd890d177c90c49b3594be39fef569b44b470417d385a85ccda" Oct 29 23:00:19.189974 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:19.189951 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm_odh-ai-gateway-infra(3f056935-3821-4d4a-8fe6-7a9e41a9c6eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" Oct 29 23:00:19.192078 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.192060 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" Oct 29 23:00:19.192953 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.192548 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp" event={"ID":"722b2177-f08d-4203-b365-310ef1c795a5","Type":"ContainerDied","Data":"0dca4bf66350183e9f3bac67f348cb6f910513933f3af4a03d2ecc8a67674cf7"} Oct 29 23:00:19.194886 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.194864 5367 generic.go:356] "Generic (PLEG): container finished" podID="fe3ae329-788f-4719-b10e-146467086b36" containerID="335b40b56383959ebea7641f5fccf9deff0c84a9c848ad1225073a113fb5cbe1" exitCode=60 Oct 29 23:00:19.197079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.196998 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerDied","Data":"335b40b56383959ebea7641f5fccf9deff0c84a9c848ad1225073a113fb5cbe1"} Oct 29 23:00:19.197967 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.197490 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerStarted","Data":"0d213b6c222d98c2f7d5a8c46828cc91295814ac8727f7e3ad13903263d340b9"} Oct 29 23:00:19.197967 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.197844 5367 scope.go:122] "RemoveContainer" containerID="335b40b56383959ebea7641f5fccf9deff0c84a9c848ad1225073a113fb5cbe1" Oct 29 23:00:19.199067 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:19.198557 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56_odh-ai-gateway-infra(fe3ae329-788f-4719-b10e-146467086b36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" podUID="fe3ae329-788f-4719-b10e-146467086b36" Oct 29 23:00:19.200002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.199609 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" Oct 29 23:00:19.200153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.199597 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m" event={"ID":"8003be9d-b7c7-4b0a-b988-02d13f22102b","Type":"ContainerDied","Data":"c750b2af1fbf1f30cca331d0b4ad3f360fc6c46739e681b3f3a2282ddf3a114d"} Oct 29 23:00:19.217264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.215468 5367 scope.go:122] "RemoveContainer" containerID="3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a" Oct 29 23:00:19.217264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.216122 5367 scope.go:122] "RemoveContainer" containerID="e16f36782ba150ef84fc7355d0bd0a7297271945e46ebecf6c0eb10a58225423" Oct 29 23:00:19.217719 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.217633 5367 scope.go:122] "RemoveContainer" containerID="fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b" Oct 29 23:00:19.219292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.218346 5367 scope.go:122] "RemoveContainer" containerID="91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1" Oct 29 23:00:19.226396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.221829 5367 scope.go:122] "RemoveContainer" containerID="f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343" Oct 29 23:00:19.229909 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.228886 5367 scope.go:122] "RemoveContainer" containerID="9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898" Oct 29 23:00:19.230036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.229924 5367 scope.go:122] "RemoveContainer" containerID="72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364" Oct 29 23:00:19.239970 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.239943 5367 generic.go:356] "Generic (PLEG): container finished" podID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerID="373dda7c65045822d698d6b68ce7b1da9a012e99ae2fdcbaa9a46bd8cf1750d0" exitCode=60 Oct 29 23:00:19.253183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.253151 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerDied","Data":"373dda7c65045822d698d6b68ce7b1da9a012e99ae2fdcbaa9a46bd8cf1750d0"} Oct 29 23:00:19.267741 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.253653 5367 scope.go:122] "RemoveContainer" containerID="373dda7c65045822d698d6b68ce7b1da9a012e99ae2fdcbaa9a46bd8cf1750d0" Oct 29 23:00:19.267741 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:19.254002 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5_odh-ai-gateway-infra(9cdf2542-f96c-4c37-88e7-fdb0a7778b43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" Oct 29 23:00:19.347085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.347057 5367 scope.go:122] "RemoveContainer" containerID="94f625fca2be71ec9e2d442786045d065bf4367098af7402af82ad6f767cd5ae" Oct 29 23:00:19.393957 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.393929 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body= Oct 29 23:00:19.394095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.393980 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" Oct 29 23:00:19.420057 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.419964 5367 scope.go:122] "RemoveContainer" containerID="549e903f6f768db9b1dab6c4f6212f585495d21a408dceef54115f8d77c39a02" Oct 29 23:00:19.502616 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.501514 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5"] Oct 29 23:00:19.618265 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.617664 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp"] Oct 29 23:00:19.627453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.626732 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a15f4-29888580-2qxmp"] Oct 29 23:00:19.637149 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.637054 5367 scope.go:122] "RemoveContainer" containerID="0aacbbac5ab399815674899d09c4bec0ce0b82c0b96335075b93dc41f4db24cc" Oct 29 23:00:19.644460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.644376 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:19.701800 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.701777 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:19.720566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.720535 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m"] Oct 29 23:00:19.730204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.730177 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm"] Oct 29 23:00:19.734538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.734519 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-92fd86-29888580-5tt9m"] Oct 29 23:00:19.747349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.747312 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca\") pod \"59a80962-cbc0-4103-a7ef-387f88af4fdb\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " Oct 29 23:00:19.748499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.747385 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk\" (UniqueName: \"kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk\") pod \"76ba151d-e166-4c58-8d4e-6e507a0986e1\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " Oct 29 23:00:19.748499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.747570 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca\") pod \"76ba151d-e166-4c58-8d4e-6e507a0986e1\" (UID: \"76ba151d-e166-4c58-8d4e-6e507a0986e1\") " Oct 29 23:00:19.748499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.748248 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b\" (UniqueName: \"kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b\") pod \"59a80962-cbc0-4103-a7ef-387f88af4fdb\" (UID: \"59a80962-cbc0-4103-a7ef-387f88af4fdb\") " Oct 29 23:00:19.748499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.748250 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca" pod "76ba151d-e166-4c58-8d4e-6e507a0986e1" (UID: "76ba151d-e166-4c58-8d4e-6e507a0986e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:19.748499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.748305 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca" pod "59a80962-cbc0-4103-a7ef-387f88af4fdb" (UID: "59a80962-cbc0-4103-a7ef-387f88af4fdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:19.748997 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.748972 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80962-cbc0-4103-a7ef-387f88af4fdb-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.749108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.749003 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ba151d-e166-4c58-8d4e-6e507a0986e1-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.788781 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.788752 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq"] Oct 29 23:00:19.825808 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.825780 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56"] Oct 29 23:00:19.894682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.888121 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk" pod "76ba151d-e166-4c58-8d4e-6e507a0986e1" (UID: "76ba151d-e166-4c58-8d4e-6e507a0986e1"). InnerVolumeSpecName "kube-api-access-929qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:19.898066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.898035 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b" pod "59a80962-cbc0-4103-a7ef-387f88af4fdb" (UID: "59a80962-cbc0-4103-a7ef-387f88af4fdb"). InnerVolumeSpecName "kube-api-access-dzt8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:19.901681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.901450 5367 scope.go:122] "RemoveContainer" containerID="02849d89e07948448ea4b19d26a436fd33371fd4223bac76e8c1a7b26c552bb4" Oct 29 23:00:19.907040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.906956 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm"] Oct 29 23:00:19.961008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.960839 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzt8b\" (UniqueName: \"kubernetes.io/projected/59a80962-cbc0-4103-a7ef-387f88af4fdb-kube-api-access-dzt8b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.961008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.960944 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-929qk\" (UniqueName: \"kubernetes.io/projected/76ba151d-e166-4c58-8d4e-6e507a0986e1-kube-api-access-929qk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.996878 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:19.996855 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:20.034381 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.034353 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm"] Oct 29 23:00:20.041592 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.041410 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de3473-p6bqm"] Oct 29 23:00:20.068378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.068346 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca\") pod \"af08130d-e1d7-41eb-abe7-f626b698231f\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " Oct 29 23:00:20.085721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.084902 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp\" (UniqueName: \"kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp\") pod \"af08130d-e1d7-41eb-abe7-f626b698231f\" (UID: \"af08130d-e1d7-41eb-abe7-f626b698231f\") " Oct 29 23:00:20.085873 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.068712 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca" pod "af08130d-e1d7-41eb-abe7-f626b698231f" (UID: "af08130d-e1d7-41eb-abe7-f626b698231f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.086322 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.086291 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af08130d-e1d7-41eb-abe7-f626b698231f-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.088828 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.088789 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp" pod "af08130d-e1d7-41eb-abe7-f626b698231f" (UID: "af08130d-e1d7-41eb-abe7-f626b698231f"). InnerVolumeSpecName "kube-api-access-79ddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.188505 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.188401 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ddp\" (UniqueName: \"kubernetes.io/projected/af08130d-e1d7-41eb-abe7-f626b698231f-kube-api-access-79ddp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.248944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.248898 5367 scope.go:122] "RemoveContainer" containerID="1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3" Oct 29 23:00:20.316930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.316906 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" Oct 29 23:00:20.317061 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.316933 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n" event={"ID":"af08130d-e1d7-41eb-abe7-f626b698231f","Type":"ContainerDied","Data":"95037ca50ad23d11c9da0bfc6e7b6807df52f971319490494c774eaf6b13501d"} Oct 29 23:00:20.317061 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.316986 5367 scope.go:122] "RemoveContainer" containerID="8c1267c19edd1ebb679d5d6413f0a6f4ef485ab942971e2f6c4cf6c7725a03f8" Oct 29 23:00:20.318677 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.318656 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" event={"ID":"59a80962-cbc0-4103-a7ef-387f88af4fdb","Type":"ContainerDied","Data":"f9d775f3ce66ad738959e9d53747da3cbdc4c6e383ea7ad80bfc5dbc264248f6"} Oct 29 23:00:20.318677 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.318672 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95" Oct 29 23:00:20.323130 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.323083 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerStarted","Data":"4733b6520dc30aea8b4b9e2cff898ff0445e3e63dbcac092a6e1797bb8cfe783"} Oct 29 23:00:20.330781 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.330758 5367 scope.go:122] "RemoveContainer" containerID="c081b1cefb8f2d41c57da9873785dbf6e8b3ae04a58b41d1fcb30dd64336cfe1" Oct 29 23:00:20.331173 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.331149 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerStarted","Data":"765620c2db32878b3ba1d278fda9f388998ad372a349d605725e66fcbf9cfdc7"} Oct 29 23:00:20.343810 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.343788 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerStarted","Data":"c92e09158d7f72cc2a764d8b4f625a975d5a6501d9077aa8a8c16ad40aca61ba"} Oct 29 23:00:20.349289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.349263 5367 generic.go:356] "Generic (PLEG): container finished" podID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerID="0d213b6c222d98c2f7d5a8c46828cc91295814ac8727f7e3ad13903263d340b9" exitCode=60 Oct 29 23:00:20.349442 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.349402 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerDied","Data":"0d213b6c222d98c2f7d5a8c46828cc91295814ac8727f7e3ad13903263d340b9"} Oct 29 23:00:20.350335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.350311 5367 scope.go:122] "RemoveContainer" containerID="45d6015d1363998864228b6bffa580b90352aecc1866579bae64887f8212b833" Oct 29 23:00:20.368722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.361187 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerStarted","Data":"cf0133ef2ee87d7218c1ada6128b2ad780546240584f62a7d0271a7fd905f553"} Oct 29 23:00:20.392779 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.392320 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body= Oct 29 23:00:20.392779 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.392376 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" Oct 29 23:00:20.401955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.401924 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerStarted","Data":"11d0c49121968926358f4f8e2bd619f6041741d08c6829bf5d8f252a3a7ec38d"} Oct 29 23:00:20.406936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.406437 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95"] Oct 29 23:00:20.412731 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.412684 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerStarted","Data":"8747f9fb3536b723e9821887ff43fd1699fe6110475240e44bf6f97ef3271b5a"} Oct 29 23:00:20.413765 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.413690 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-329f7b-29888580-xts95"] Oct 29 23:00:20.415427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.415410 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" event={"ID":"76ba151d-e166-4c58-8d4e-6e507a0986e1","Type":"ContainerDied","Data":"081727edfb397d37b08051f9b17d03e2f3480b02fc01324eae4cf3890f3f9d11"} Oct 29 23:00:20.415598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.415586 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm" Oct 29 23:00:20.421244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.421227 5367 scope.go:122] "RemoveContainer" containerID="14629b1ce6105c3002cb34ec1a1e5e6201557e553fc69cc7cfe184846fd7ee09" Oct 29 23:00:20.438388 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.438356 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n"] Oct 29 23:00:20.445765 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.445717 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888580-kvs7n"] Oct 29 23:00:20.827356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.827297 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm"] Oct 29 23:00:20.835288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.835265 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888580-lwmlm"] Oct 29 23:00:20.983823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.983744 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:20.986830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.986805 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:20.991559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.991532 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:20.996224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:20.996205 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:21.001195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.001178 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:21.035153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035124 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca\") pod \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " Oct 29 23:00:21.035347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035287 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw\" (UniqueName: \"kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw\") pod \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\" (UID: \"9cdf2542-f96c-4c37-88e7-fdb0a7778b43\") " Oct 29 23:00:21.035347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035329 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca\") pod \"fbdec272-136f-4d21-a496-fe04aede5621\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " Oct 29 23:00:21.035472 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035367 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5\" (UniqueName: \"kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5\") pod \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " Oct 29 23:00:21.035472 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035417 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca\") pod \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\" (UID: \"5ee465e5-6f19-4d52-9e85-00e40b15bd06\") " Oct 29 23:00:21.035582 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035463 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca" pod "9cdf2542-f96c-4c37-88e7-fdb0a7778b43" (UID: "9cdf2542-f96c-4c37-88e7-fdb0a7778b43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:21.035582 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035490 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2\" (UniqueName: \"kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2\") pod \"fe3ae329-788f-4719-b10e-146467086b36\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " Oct 29 23:00:21.035582 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035524 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca\") pod \"fe3ae329-788f-4719-b10e-146467086b36\" (UID: \"fe3ae329-788f-4719-b10e-146467086b36\") " Oct 29 23:00:21.035582 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035569 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca\") pod \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " Oct 29 23:00:21.035820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035656 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq\" (UniqueName: \"kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq\") pod \"fbdec272-136f-4d21-a496-fe04aede5621\" (UID: \"fbdec272-136f-4d21-a496-fe04aede5621\") " Oct 29 23:00:21.035820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035745 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b\") pod \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\" (UID: \"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb\") " Oct 29 23:00:21.035820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035804 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca" pod "fbdec272-136f-4d21-a496-fe04aede5621" (UID: "fbdec272-136f-4d21-a496-fe04aede5621"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:21.035977 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.035869 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca" pod "5ee465e5-6f19-4d52-9e85-00e40b15bd06" (UID: "5ee465e5-6f19-4d52-9e85-00e40b15bd06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:21.036439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.036413 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca" pod "fe3ae329-788f-4719-b10e-146467086b36" (UID: "fe3ae329-788f-4719-b10e-146467086b36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:21.036609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.036589 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.036668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.036617 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdec272-136f-4d21-a496-fe04aede5621-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.036668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.036633 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee465e5-6f19-4d52-9e85-00e40b15bd06-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.036668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.036647 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3ae329-788f-4719-b10e-146467086b36-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.036866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.036842 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca" pod "3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" (UID: "3f056935-3821-4d4a-8fe6-7a9e41a9c6eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:21.038033 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.038010 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw" pod "9cdf2542-f96c-4c37-88e7-fdb0a7778b43" (UID: "9cdf2542-f96c-4c37-88e7-fdb0a7778b43"). InnerVolumeSpecName "kube-api-access-sp2rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:21.038437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.038419 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b" pod "3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" (UID: "3f056935-3821-4d4a-8fe6-7a9e41a9c6eb"). InnerVolumeSpecName "kube-api-access-pmz8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:21.038529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.038491 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2" pod "fe3ae329-788f-4719-b10e-146467086b36" (UID: "fe3ae329-788f-4719-b10e-146467086b36"). InnerVolumeSpecName "kube-api-access-dmmr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:21.038589 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.038575 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq" pod "fbdec272-136f-4d21-a496-fe04aede5621" (UID: "fbdec272-136f-4d21-a496-fe04aede5621"). InnerVolumeSpecName "kube-api-access-wg7zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:21.039497 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.039475 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5" pod "5ee465e5-6f19-4d52-9e85-00e40b15bd06" (UID: "5ee465e5-6f19-4d52-9e85-00e40b15bd06"). InnerVolumeSpecName "kube-api-access-knpj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:21.138387 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.138352 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmr2\" (UniqueName: \"kubernetes.io/projected/fe3ae329-788f-4719-b10e-146467086b36-kube-api-access-dmmr2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.138387 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.138390 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.138612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.138407 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg7zq\" (UniqueName: \"kubernetes.io/projected/fbdec272-136f-4d21-a496-fe04aede5621-kube-api-access-wg7zq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.138612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.138422 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb-kube-api-access-pmz8b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.138612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.138436 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2rw\" (UniqueName: \"kubernetes.io/projected/9cdf2542-f96c-4c37-88e7-fdb0a7778b43-kube-api-access-sp2rw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.138612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.138451 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knpj5\" (UniqueName: \"kubernetes.io/projected/5ee465e5-6f19-4d52-9e85-00e40b15bd06-kube-api-access-knpj5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.234349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.234246 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" path="/var/lib/kubelet/pods/59a80962-cbc0-4103-a7ef-387f88af4fdb/volumes" Oct 29 23:00:21.235751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.235724 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722b2177-f08d-4203-b365-310ef1c795a5" path="/var/lib/kubelet/pods/722b2177-f08d-4203-b365-310ef1c795a5/volumes" Oct 29 23:00:21.237765 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.237272 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" path="/var/lib/kubelet/pods/76ba151d-e166-4c58-8d4e-6e507a0986e1/volumes" Oct 29 23:00:21.249006 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.248976 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" path="/var/lib/kubelet/pods/8003be9d-b7c7-4b0a-b988-02d13f22102b/volumes" Oct 29 23:00:21.249771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.249748 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" path="/var/lib/kubelet/pods/af08130d-e1d7-41eb-abe7-f626b698231f/volumes" Oct 29 23:00:21.254564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.253259 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf12c357-05c8-4ec6-bbea-e74dfca774d3" path="/var/lib/kubelet/pods/cf12c357-05c8-4ec6-bbea-e74dfca774d3/volumes" Oct 29 23:00:21.431416 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.431386 5367 generic.go:356] "Generic (PLEG): container finished" podID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerID="4733b6520dc30aea8b4b9e2cff898ff0445e3e63dbcac092a6e1797bb8cfe783" exitCode=60 Oct 29 23:00:21.431560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.431469 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerDied","Data":"4733b6520dc30aea8b4b9e2cff898ff0445e3e63dbcac092a6e1797bb8cfe783"} Oct 29 23:00:21.431560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.431545 5367 scope.go:122] "RemoveContainer" containerID="fd320e3d5ccc246709dafb497c66d04827966aaa69b509235d8300fd2717dd7b" Oct 29 23:00:21.432146 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.432127 5367 scope.go:122] "RemoveContainer" containerID="4733b6520dc30aea8b4b9e2cff898ff0445e3e63dbcac092a6e1797bb8cfe783" Oct 29 23:00:21.432627 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:21.432605 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj_odh-ai-gateway-infra(def8b4aa-4cbf-43d5-ac35-43a87301a566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" Oct 29 23:00:21.434912 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.434888 5367 generic.go:356] "Generic (PLEG): container finished" podID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerID="765620c2db32878b3ba1d278fda9f388998ad372a349d605725e66fcbf9cfdc7" exitCode=60 Oct 29 23:00:21.435029 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.434958 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerDied","Data":"765620c2db32878b3ba1d278fda9f388998ad372a349d605725e66fcbf9cfdc7"} Oct 29 23:00:21.435309 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.435295 5367 scope.go:122] "RemoveContainer" containerID="765620c2db32878b3ba1d278fda9f388998ad372a349d605725e66fcbf9cfdc7" Oct 29 23:00:21.435689 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:21.435668 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn_odh-ai-gateway-infra(f64ffbae-aa0d-4478-8a93-07c0094d7c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" Oct 29 23:00:21.436546 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.436531 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" Oct 29 23:00:21.436546 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.436536 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm" event={"ID":"3f056935-3821-4d4a-8fe6-7a9e41a9c6eb","Type":"ContainerDied","Data":"f027b1b596032790e678fcad745025a3d6dd87a47f5c705fb5c55232cb41674a"} Oct 29 23:00:21.438880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.438854 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerStarted","Data":"64beb4b23b13ed098f5a4192a45d73659564246e0e0e97946c3a73ad03102e0e"} Oct 29 23:00:21.440267 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.440251 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" Oct 29 23:00:21.440369 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.440271 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56" event={"ID":"fe3ae329-788f-4719-b10e-146467086b36","Type":"ContainerDied","Data":"fd8305d3c4e1778356ef363ba34e867be48727b41123fac7188013023b82e011"} Oct 29 23:00:21.442205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.442188 5367 generic.go:356] "Generic (PLEG): container finished" podID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerID="c92e09158d7f72cc2a764d8b4f625a975d5a6501d9077aa8a8c16ad40aca61ba" exitCode=60 Oct 29 23:00:21.442289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.442237 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerDied","Data":"c92e09158d7f72cc2a764d8b4f625a975d5a6501d9077aa8a8c16ad40aca61ba"} Oct 29 23:00:21.442478 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.442465 5367 scope.go:122] "RemoveContainer" containerID="c92e09158d7f72cc2a764d8b4f625a975d5a6501d9077aa8a8c16ad40aca61ba" Oct 29 23:00:21.442692 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:21.442673 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp_odh-ai-gateway-infra(7531bdb4-381a-4039-bcea-5ed7379a8546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" Oct 29 23:00:21.444282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.444221 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" Oct 29 23:00:21.444282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.444263 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm" event={"ID":"5ee465e5-6f19-4d52-9e85-00e40b15bd06","Type":"ContainerDied","Data":"cdf26b0e32700293167463217c7d093911f6dca5eaa2d58b7bf48b7f4a3e09e8"} Oct 29 23:00:21.446266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.446247 5367 generic.go:356] "Generic (PLEG): container finished" podID="d119d843-8ea0-444b-9590-614da9a4f435" containerID="cf0133ef2ee87d7218c1ada6128b2ad780546240584f62a7d0271a7fd905f553" exitCode=60 Oct 29 23:00:21.446360 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.446305 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerDied","Data":"cf0133ef2ee87d7218c1ada6128b2ad780546240584f62a7d0271a7fd905f553"} Oct 29 23:00:21.446662 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.446641 5367 scope.go:122] "RemoveContainer" containerID="cf0133ef2ee87d7218c1ada6128b2ad780546240584f62a7d0271a7fd905f553" Oct 29 23:00:21.446938 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:21.446902 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7_odh-ai-gateway-infra(d119d843-8ea0-444b-9590-614da9a4f435)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" podUID="d119d843-8ea0-444b-9590-614da9a4f435" Oct 29 23:00:21.448437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.448415 5367 generic.go:356] "Generic (PLEG): container finished" podID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerID="11d0c49121968926358f4f8e2bd619f6041741d08c6829bf5d8f252a3a7ec38d" exitCode=60 Oct 29 23:00:21.448594 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.448519 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerDied","Data":"11d0c49121968926358f4f8e2bd619f6041741d08c6829bf5d8f252a3a7ec38d"} Oct 29 23:00:21.448922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.448902 5367 scope.go:122] "RemoveContainer" containerID="11d0c49121968926358f4f8e2bd619f6041741d08c6829bf5d8f252a3a7ec38d" Oct 29 23:00:21.449190 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:21.449165 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5_odh-ai-gateway-infra(9ba7d901-6282-41be-ad0f-fd74055bad81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" Oct 29 23:00:21.450076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.450061 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" Oct 29 23:00:21.450255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.450234 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5" event={"ID":"9cdf2542-f96c-4c37-88e7-fdb0a7778b43","Type":"ContainerDied","Data":"7b791374e729d328135689af9dfcb41e8c088157c1da1b13d6b8f56c178959de"} Oct 29 23:00:21.452801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.452781 5367 generic.go:356] "Generic (PLEG): container finished" podID="81935cce-192e-4569-afe7-b61fe0a9a419" containerID="8747f9fb3536b723e9821887ff43fd1699fe6110475240e44bf6f97ef3271b5a" exitCode=60 Oct 29 23:00:21.452927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.452855 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerDied","Data":"8747f9fb3536b723e9821887ff43fd1699fe6110475240e44bf6f97ef3271b5a"} Oct 29 23:00:21.453217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.453194 5367 scope.go:122] "RemoveContainer" containerID="8747f9fb3536b723e9821887ff43fd1699fe6110475240e44bf6f97ef3271b5a" Oct 29 23:00:21.453513 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:21.453449 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb_odh-ai-gateway-infra(81935cce-192e-4569-afe7-b61fe0a9a419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" Oct 29 23:00:21.458547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.458526 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" Oct 29 23:00:21.458635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.458540 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq" event={"ID":"fbdec272-136f-4d21-a496-fe04aede5621","Type":"ContainerDied","Data":"2191c882a5d3210d2c1af70a48c766dba189e867c3331ba4e51fe7fb36c65ada"} Oct 29 23:00:21.459306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.459291 5367 scope.go:122] "RemoveContainer" containerID="f1c0da71967a2c033ee57668df370913cddc210afeae54dbef3f7cbd75356343" Oct 29 23:00:21.489330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.489230 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:21.490001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.489981 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:21.491107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.491088 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:00:21.520486 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.520461 5367 scope.go:122] "RemoveContainer" containerID="267bd81ad6ee2dd890d177c90c49b3594be39fef569b44b470417d385a85ccda" Oct 29 23:00:21.532215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.532194 5367 scope.go:122] "RemoveContainer" containerID="335b40b56383959ebea7641f5fccf9deff0c84a9c848ad1225073a113fb5cbe1" Oct 29 23:00:21.536400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.536380 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj"] Oct 29 23:00:21.545013 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.544992 5367 scope.go:122] "RemoveContainer" containerID="9bc18a8003c3e3750cace5eba338580aa67db7ca820771d03726aaa9f9958898" Oct 29 23:00:21.561537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.561515 5367 scope.go:122] "RemoveContainer" containerID="0d213b6c222d98c2f7d5a8c46828cc91295814ac8727f7e3ad13903263d340b9" Oct 29 23:00:21.576385 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.576367 5367 scope.go:122] "RemoveContainer" containerID="72d3ed1153bddffd45a511f6d1b2546b6c432efb8e333b7f370734c8d5d00364" Oct 29 23:00:21.591576 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.591557 5367 scope.go:122] "RemoveContainer" containerID="3dd33d5ad605b115ab26c6dd491eb71815a97c9d87d2ade4a6cf225e7436e31a" Oct 29 23:00:21.606949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.606925 5367 scope.go:122] "RemoveContainer" containerID="373dda7c65045822d698d6b68ce7b1da9a012e99ae2fdcbaa9a46bd8cf1750d0" Oct 29 23:00:21.616748 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.616728 5367 scope.go:122] "RemoveContainer" containerID="91fa9a60f4a2fd9023c0aee23441f29378fa6210cbc4b0de881e72d6301660a1" Oct 29 23:00:21.630858 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.630732 5367 scope.go:122] "RemoveContainer" containerID="0c7840c42080716e9417c22c48f4f507596513fdc3a00fb5df3ab596a3644ef6" Oct 29 23:00:21.638051 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.638031 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn"] Oct 29 23:00:21.773144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.773069 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7"] Oct 29 23:00:21.915634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.915599 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm"] Oct 29 23:00:21.923002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.922970 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888580-8sxpm"] Oct 29 23:00:21.928447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.928422 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp"] Oct 29 23:00:21.932812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.932786 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5"] Oct 29 23:00:21.938226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.938202 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2ae845-29888580-9x7k5"] Oct 29 23:00:21.977362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.977336 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm"] Oct 29 23:00:21.981532 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:21.981509 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888580-p8bhm"] Oct 29 23:00:22.018789 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.018686 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb"] Oct 29 23:00:22.023374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.023313 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq"] Oct 29 23:00:22.027415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.027393 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888580-zdgqq"] Oct 29 23:00:22.061944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.061920 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56"] Oct 29 23:00:22.066091 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.066068 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dabb8-29888580-mrp56"] Oct 29 23:00:22.085851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.085822 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" podUID="5f2194a3-766d-48a4-8c3a-2eccc7e417ab" containerName="payload-pre-processing" containerID="cri-o://9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca" gracePeriod=25 Oct 29 23:00:22.119618 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.119592 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5"] Oct 29 23:00:22.339051 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:22.339012 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:00:22.377062 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.377038 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 23:00:22.477559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.477525 5367 generic.go:356] "Generic (PLEG): container finished" podID="5f2194a3-766d-48a4-8c3a-2eccc7e417ab" containerID="9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca" exitCode=0 Oct 29 23:00:22.477683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.477558 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" Oct 29 23:00:22.477683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.477671 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" event={"ID":"5f2194a3-766d-48a4-8c3a-2eccc7e417ab","Type":"ContainerDied","Data":"9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca"} Oct 29 23:00:22.477855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.477766 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd" event={"ID":"5f2194a3-766d-48a4-8c3a-2eccc7e417ab","Type":"ContainerDied","Data":"e1439111ba1d223391d883dec06bfec7d564c069678381c222556026329db3c1"} Oct 29 23:00:22.477855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.477803 5367 scope.go:122] "RemoveContainer" containerID="9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca" Oct 29 23:00:22.486414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.486392 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume\") pod \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " Oct 29 23:00:22.486567 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.486466 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk\" (UniqueName: \"kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk\") pod \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\" (UID: \"5f2194a3-766d-48a4-8c3a-2eccc7e417ab\") " Oct 29 23:00:22.487270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.486963 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume" pod "5f2194a3-766d-48a4-8c3a-2eccc7e417ab" (UID: "5f2194a3-766d-48a4-8c3a-2eccc7e417ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:22.487270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.487192 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.499017 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.498916 5367 scope.go:122] "RemoveContainer" containerID="9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca" Oct 29 23:00:22.499691 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:22.499608 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca\": container with ID starting with 9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca not found: ID does not exist" containerID="9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca" Oct 29 23:00:22.499691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.499650 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca"} err="failed to get container status \"9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca\": rpc error: code = NotFound desc = could not find container \"9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca\": container with ID starting with 9042c8a3aaabbc7731e827aa43b44ce3deed44a222e307d9ab9ed26b4ad816ca not found: ID does not exist" Oct 29 23:00:22.503342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.501929 5367 generic.go:356] "Generic (PLEG): container finished" podID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerID="64beb4b23b13ed098f5a4192a45d73659564246e0e0e97946c3a73ad03102e0e" exitCode=60 Oct 29 23:00:22.503342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.502022 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerDied","Data":"64beb4b23b13ed098f5a4192a45d73659564246e0e0e97946c3a73ad03102e0e"} Oct 29 23:00:22.503342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.502072 5367 scope.go:122] "RemoveContainer" containerID="1fcb8e0a651cb79a1f2e16af39be5534754a3d0d2545df807ac94651b6c12cc3" Oct 29 23:00:22.503342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.502542 5367 scope.go:122] "RemoveContainer" containerID="64beb4b23b13ed098f5a4192a45d73659564246e0e0e97946c3a73ad03102e0e" Oct 29 23:00:22.503342 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:22.502977 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk_odh-ai-gateway-infra(ac6c78c9-5de2-4bc5-91e5-9f891eb598b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" Oct 29 23:00:22.517650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.517530 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk" pod "5f2194a3-766d-48a4-8c3a-2eccc7e417ab" (UID: "5f2194a3-766d-48a4-8c3a-2eccc7e417ab"). InnerVolumeSpecName "kube-api-access-xw4rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:22.600906 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.589274 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4rk\" (UniqueName: \"kubernetes.io/projected/5f2194a3-766d-48a4-8c3a-2eccc7e417ab-kube-api-access-xw4rk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.746746 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.746633 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:22.774548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.774522 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk"] Oct 29 23:00:22.836500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.836144 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:22.852658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.852602 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:22.854790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.854768 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:22.859558 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.859540 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:22.865474 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.865455 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:22.906242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.906205 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca\") pod \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " Oct 29 23:00:22.906414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.906299 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj\" (UniqueName: \"kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj\") pod \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\" (UID: \"f64ffbae-aa0d-4478-8a93-07c0094d7c63\") " Oct 29 23:00:22.906656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.906632 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca" pod "f64ffbae-aa0d-4478-8a93-07c0094d7c63" (UID: "f64ffbae-aa0d-4478-8a93-07c0094d7c63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:22.906880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.906862 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca\") pod \"d119d843-8ea0-444b-9590-614da9a4f435\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " Oct 29 23:00:22.906955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.906933 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd\" (UniqueName: \"kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd\") pod \"d119d843-8ea0-444b-9590-614da9a4f435\" (UID: \"d119d843-8ea0-444b-9590-614da9a4f435\") " Oct 29 23:00:22.907293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.907268 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca" pod "d119d843-8ea0-444b-9590-614da9a4f435" (UID: "d119d843-8ea0-444b-9590-614da9a4f435"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:22.907754 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.907692 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64ffbae-aa0d-4478-8a93-07c0094d7c63-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.907881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.907761 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d119d843-8ea0-444b-9590-614da9a4f435-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.909141 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.909109 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj" pod "f64ffbae-aa0d-4478-8a93-07c0094d7c63" (UID: "f64ffbae-aa0d-4478-8a93-07c0094d7c63"). InnerVolumeSpecName "kube-api-access-fx8fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:22.909485 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.909461 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd" pod "d119d843-8ea0-444b-9590-614da9a4f435" (UID: "d119d843-8ea0-444b-9590-614da9a4f435"). InnerVolumeSpecName "kube-api-access-jcdfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:22.929767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.929547 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd"] Oct 29 23:00:22.934423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:22.934401 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6df6bdc895-f7rhd"] Oct 29 23:00:23.009872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.009835 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca\") pod \"81935cce-192e-4569-afe7-b61fe0a9a419\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " Oct 29 23:00:23.010101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.009913 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca\") pod \"7531bdb4-381a-4039-bcea-5ed7379a8546\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " Oct 29 23:00:23.010101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.010084 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56\" (UniqueName: \"kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56\") pod \"7531bdb4-381a-4039-bcea-5ed7379a8546\" (UID: \"7531bdb4-381a-4039-bcea-5ed7379a8546\") " Oct 29 23:00:23.010297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.010141 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca\") pod \"9ba7d901-6282-41be-ad0f-fd74055bad81\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " Oct 29 23:00:23.010297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.010219 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq\" (UniqueName: \"kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq\") pod \"81935cce-192e-4569-afe7-b61fe0a9a419\" (UID: \"81935cce-192e-4569-afe7-b61fe0a9a419\") " Oct 29 23:00:23.010462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.010296 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn\" (UniqueName: \"kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn\") pod \"def8b4aa-4cbf-43d5-ac35-43a87301a566\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " Oct 29 23:00:23.010462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.010375 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5\" (UniqueName: \"kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5\") pod \"9ba7d901-6282-41be-ad0f-fd74055bad81\" (UID: \"9ba7d901-6282-41be-ad0f-fd74055bad81\") " Oct 29 23:00:23.010639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.010514 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca\") pod \"def8b4aa-4cbf-43d5-ac35-43a87301a566\" (UID: \"def8b4aa-4cbf-43d5-ac35-43a87301a566\") " Oct 29 23:00:23.011309 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.011271 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8fj\" (UniqueName: \"kubernetes.io/projected/f64ffbae-aa0d-4478-8a93-07c0094d7c63-kube-api-access-fx8fj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.011309 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.011304 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdfd\" (UniqueName: \"kubernetes.io/projected/d119d843-8ea0-444b-9590-614da9a4f435-kube-api-access-jcdfd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.011648 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.011279 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca" pod "81935cce-192e-4569-afe7-b61fe0a9a419" (UID: "81935cce-192e-4569-afe7-b61fe0a9a419"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:23.011791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.011644 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca" pod "def8b4aa-4cbf-43d5-ac35-43a87301a566" (UID: "def8b4aa-4cbf-43d5-ac35-43a87301a566"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:23.012012 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.011987 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca" pod "7531bdb4-381a-4039-bcea-5ed7379a8546" (UID: "7531bdb4-381a-4039-bcea-5ed7379a8546"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:23.012935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.012889 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca" pod "9ba7d901-6282-41be-ad0f-fd74055bad81" (UID: "9ba7d901-6282-41be-ad0f-fd74055bad81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:23.015390 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.015365 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56" pod "7531bdb4-381a-4039-bcea-5ed7379a8546" (UID: "7531bdb4-381a-4039-bcea-5ed7379a8546"). InnerVolumeSpecName "kube-api-access-kxv56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:23.016761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.016694 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn" pod "def8b4aa-4cbf-43d5-ac35-43a87301a566" (UID: "def8b4aa-4cbf-43d5-ac35-43a87301a566"). InnerVolumeSpecName "kube-api-access-66grn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:23.018167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.018140 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq" pod "81935cce-192e-4569-afe7-b61fe0a9a419" (UID: "81935cce-192e-4569-afe7-b61fe0a9a419"). InnerVolumeSpecName "kube-api-access-njsbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:23.018289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.018266 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5" pod "9ba7d901-6282-41be-ad0f-fd74055bad81" (UID: "9ba7d901-6282-41be-ad0f-fd74055bad81"). InnerVolumeSpecName "kube-api-access-4pzv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:23.113509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113429 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def8b4aa-4cbf-43d5-ac35-43a87301a566-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113459 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81935cce-192e-4569-afe7-b61fe0a9a419-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113471 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7531bdb4-381a-4039-bcea-5ed7379a8546-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113483 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxv56\" (UniqueName: \"kubernetes.io/projected/7531bdb4-381a-4039-bcea-5ed7379a8546-kube-api-access-kxv56\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113494 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba7d901-6282-41be-ad0f-fd74055bad81-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113506 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njsbq\" (UniqueName: \"kubernetes.io/projected/81935cce-192e-4569-afe7-b61fe0a9a419-kube-api-access-njsbq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113520 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66grn\" (UniqueName: \"kubernetes.io/projected/def8b4aa-4cbf-43d5-ac35-43a87301a566-kube-api-access-66grn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.113871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.113534 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzv5\" (UniqueName: \"kubernetes.io/projected/9ba7d901-6282-41be-ad0f-fd74055bad81-kube-api-access-4pzv5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.228647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.228611 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" path="/var/lib/kubelet/pods/3f056935-3821-4d4a-8fe6-7a9e41a9c6eb/volumes" Oct 29 23:00:23.229131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.229115 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" path="/var/lib/kubelet/pods/5ee465e5-6f19-4d52-9e85-00e40b15bd06/volumes" Oct 29 23:00:23.229515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.229503 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2194a3-766d-48a4-8c3a-2eccc7e417ab" path="/var/lib/kubelet/pods/5f2194a3-766d-48a4-8c3a-2eccc7e417ab/volumes" Oct 29 23:00:23.230227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.230210 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" path="/var/lib/kubelet/pods/9cdf2542-f96c-4c37-88e7-fdb0a7778b43/volumes" Oct 29 23:00:23.230731 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.230695 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdec272-136f-4d21-a496-fe04aede5621" path="/var/lib/kubelet/pods/fbdec272-136f-4d21-a496-fe04aede5621/volumes" Oct 29 23:00:23.231160 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.231146 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3ae329-788f-4719-b10e-146467086b36" path="/var/lib/kubelet/pods/fe3ae329-788f-4719-b10e-146467086b36/volumes" Oct 29 23:00:23.528250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.528218 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" Oct 29 23:00:23.528780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.528218 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7" event={"ID":"d119d843-8ea0-444b-9590-614da9a4f435","Type":"ContainerDied","Data":"01dd03301dc721c541a0f4d23521197c62fc8c009f69e7d1c7bee1e8993e1040"} Oct 29 23:00:23.528780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.528377 5367 scope.go:122] "RemoveContainer" containerID="cf0133ef2ee87d7218c1ada6128b2ad780546240584f62a7d0271a7fd905f553" Oct 29 23:00:23.529945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.529920 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" event={"ID":"9ba7d901-6282-41be-ad0f-fd74055bad81","Type":"ContainerDied","Data":"78f39c382863d5cd4046327ecebbc698ffafcbfbe037961c70306dd10b038bd8"} Oct 29 23:00:23.530019 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.529936 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5" Oct 29 23:00:23.536715 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.534302 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" Oct 29 23:00:23.536715 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.534355 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb" event={"ID":"81935cce-192e-4569-afe7-b61fe0a9a419","Type":"ContainerDied","Data":"df0f2fdebfc25c65cf530209760aafa49a343522e44c9bdbeebd171495e06e71"} Oct 29 23:00:23.536715 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.536261 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" Oct 29 23:00:23.536715 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.536280 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj" event={"ID":"def8b4aa-4cbf-43d5-ac35-43a87301a566","Type":"ContainerDied","Data":"ac31a9d783c9ed017f428b6e14c74059d9aa09236c17550123b1cb831863c63d"} Oct 29 23:00:23.538089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.538069 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" Oct 29 23:00:23.538201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.538064 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn" event={"ID":"f64ffbae-aa0d-4478-8a93-07c0094d7c63","Type":"ContainerDied","Data":"4de4d60d695978dc8ed21b9fffd692eb5f38b51ceeb2afa9ee8c2f0550a1e4a3"} Oct 29 23:00:23.543005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.542980 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" event={"ID":"7531bdb4-381a-4039-bcea-5ed7379a8546","Type":"ContainerDied","Data":"12e6d0c0e9472a8755c3275351133c44d2ceaf0ffbb05ca65bc72f0e25c60f77"} Oct 29 23:00:23.543111 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.543033 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp" Oct 29 23:00:23.544381 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.544357 5367 scope.go:122] "RemoveContainer" containerID="11d0c49121968926358f4f8e2bd619f6041741d08c6829bf5d8f252a3a7ec38d" Oct 29 23:00:23.562185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.561998 5367 scope.go:122] "RemoveContainer" containerID="8747f9fb3536b723e9821887ff43fd1699fe6110475240e44bf6f97ef3271b5a" Oct 29 23:00:23.574051 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.574030 5367 scope.go:122] "RemoveContainer" containerID="4733b6520dc30aea8b4b9e2cff898ff0445e3e63dbcac092a6e1797bb8cfe783" Oct 29 23:00:23.587233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.587203 5367 scope.go:122] "RemoveContainer" containerID="765620c2db32878b3ba1d278fda9f388998ad372a349d605725e66fcbf9cfdc7" Oct 29 23:00:23.591591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.591513 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5"] Oct 29 23:00:23.599078 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.599051 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888580-vv2f5"] Oct 29 23:00:23.606890 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.606867 5367 scope.go:122] "RemoveContainer" containerID="c92e09158d7f72cc2a764d8b4f625a975d5a6501d9077aa8a8c16ad40aca61ba" Oct 29 23:00:23.642435 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.642406 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7"] Oct 29 23:00:23.649648 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.649626 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bcb810-29888580-gfqd7"] Oct 29 23:00:23.679450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.679422 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj"] Oct 29 23:00:23.688113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.688081 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e803b6-29888580-hl2tj"] Oct 29 23:00:23.842726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.842646 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn"] Oct 29 23:00:23.881756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.871650 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:23.918821 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.918747 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888580-d89qn"] Oct 29 23:00:23.987580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:23.987506 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb"] Oct 29 23:00:24.004599 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.004532 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-437981-29888580-pnhfb"] Oct 29 23:00:24.018955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.018926 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp"] Oct 29 23:00:24.024364 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.024337 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-95672c-29888580-zbfrp"] Oct 29 23:00:24.046192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.046116 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca\") pod \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " Oct 29 23:00:24.046553 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.046529 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z\" (UniqueName: \"kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z\") pod \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\" (UID: \"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4\") " Oct 29 23:00:24.046926 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.046565 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca" pod "ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" (UID: "ac6c78c9-5de2-4bc5-91e5-9f891eb598b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:24.047577 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.047557 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.049344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.049321 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z" pod "ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" (UID: "ac6c78c9-5de2-4bc5-91e5-9f891eb598b4"). InnerVolumeSpecName "kube-api-access-2k84z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:24.150016 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.149961 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k84z\" (UniqueName: \"kubernetes.io/projected/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4-kube-api-access-2k84z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.562601 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.562574 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" Oct 29 23:00:24.563140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.562600 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk" event={"ID":"ac6c78c9-5de2-4bc5-91e5-9f891eb598b4","Type":"ContainerDied","Data":"c45ccec23d4da46e9d9c833cf3589856d4c9edb915e675aa016cbb7c2eb56962"} Oct 29 23:00:24.563140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.562663 5367 scope.go:122] "RemoveContainer" containerID="64beb4b23b13ed098f5a4192a45d73659564246e0e0e97946c3a73ad03102e0e" Oct 29 23:00:24.619594 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.619571 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk"] Oct 29 23:00:24.626119 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:24.626088 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6f83c-29888580-vg7bk"] Oct 29 23:00:25.226063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.226036 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" path="/var/lib/kubelet/pods/7531bdb4-381a-4039-bcea-5ed7379a8546/volumes" Oct 29 23:00:25.226521 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.226507 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" path="/var/lib/kubelet/pods/81935cce-192e-4569-afe7-b61fe0a9a419/volumes" Oct 29 23:00:25.227037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.227022 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" path="/var/lib/kubelet/pods/9ba7d901-6282-41be-ad0f-fd74055bad81/volumes" Oct 29 23:00:25.227746 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.227731 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" path="/var/lib/kubelet/pods/ac6c78c9-5de2-4bc5-91e5-9f891eb598b4/volumes" Oct 29 23:00:25.228189 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.228174 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d119d843-8ea0-444b-9590-614da9a4f435" path="/var/lib/kubelet/pods/d119d843-8ea0-444b-9590-614da9a4f435/volumes" Oct 29 23:00:25.228611 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.228597 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" path="/var/lib/kubelet/pods/def8b4aa-4cbf-43d5-ac35-43a87301a566/volumes" Oct 29 23:00:25.229335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:25.229319 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" path="/var/lib/kubelet/pods/f64ffbae-aa0d-4478-8a93-07c0094d7c63/volumes" Oct 29 23:00:30.033373 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:30.033338 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f"] Oct 29 23:00:30.046447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:30.046411 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63407e7b-fd52f"] Oct 29 23:00:31.223466 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:31.223426 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd28b9e5-6491-4e2b-bc1d-29583c298b48" path="/var/lib/kubelet/pods/fd28b9e5-6491-4e2b-bc1d-29583c298b48/volumes" Oct 29 23:00:32.442690 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:32.442650 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv"] Oct 29 23:00:32.443251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:32.442948 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" containerID="cri-o://98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f" gracePeriod=30 Oct 29 23:00:33.072247 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072213 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n"] Oct 29 23:00:33.072936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072919 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072936 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072947 5367 cpu_manager.go:414] "containerMap: removing container" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072952 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072964 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072970 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072976 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072981 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072988 5367 cpu_manager.go:414] "containerMap: removing container" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.072993 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.073003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073003 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073010 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073018 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073024 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073038 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5f2194a3-766d-48a4-8c3a-2eccc7e417ab" containerName="payload-pre-processing" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073044 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2194a3-766d-48a4-8c3a-2eccc7e417ab" containerName="payload-pre-processing" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073050 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073054 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073060 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073064 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073069 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073073 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073081 5367 cpu_manager.go:414] "containerMap: removing container" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073086 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073095 5367 cpu_manager.go:414] "containerMap: removing container" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073099 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073107 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073112 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073119 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073123 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073130 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073135 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073141 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073146 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073153 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073157 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073164 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073168 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073175 5367 cpu_manager.go:414] "containerMap: removing container" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073180 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073192 5367 cpu_manager.go:414] "containerMap: removing container" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073197 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073203 5367 cpu_manager.go:414] "containerMap: removing container" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073207 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073215 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073219 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073224 5367 cpu_manager.go:414] "containerMap: removing container" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073228 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073233 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073237 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073243 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073246 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073256 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073259 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073266 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073270 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073278 5367 cpu_manager.go:414] "containerMap: removing container" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073283 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073291 5367 cpu_manager.go:414] "containerMap: removing container" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073296 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073303 5367 cpu_manager.go:414] "containerMap: removing container" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073308 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073317 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.073300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073321 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073330 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073335 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073340 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073344 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073354 5367 cpu_manager.go:414] "containerMap: removing container" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073358 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073365 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073369 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073376 5367 cpu_manager.go:414] "containerMap: removing container" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073380 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073387 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073391 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073501 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073506 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073513 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073518 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073524 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073529 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073533 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073539 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073544 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073548 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073553 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073558 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073564 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073568 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073573 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073579 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073586 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073592 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073596 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073601 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2194a3-766d-48a4-8c3a-2eccc7e417ab" containerName="payload-pre-processing" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073607 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073612 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073618 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073623 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073629 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073634 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073639 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073644 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c78c9-5de2-4bc5-91e5-9f891eb598b4" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073649 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073655 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073660 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073666 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073671 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073676 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073681 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073687 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073694 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073710 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073716 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073721 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073726 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:33.074928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.073732 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:33.077483 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.077465 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.099422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.099397 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n"] Oct 29 23:00:33.106767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.106740 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:00:33.106941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.106920 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:00:33.141572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.141548 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:33.141749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.141724 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:00:33.210277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.210245 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.210502 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.210463 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.210617 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.210603 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk7w\" (UniqueName: \"kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.321077 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.321038 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.321339 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.321310 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.321618 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.321596 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk7w\" (UniqueName: \"kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.337142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.337083 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:00:33.337244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.337209 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:33.341725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.341682 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.343911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.343886 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.356323 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.356298 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:00:33.364445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.364417 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk7w\" (UniqueName: \"kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.407137 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.407108 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:00:33.412372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.412348 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:33.720497 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:33.720460 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81e3ce7_b41b_4666_bb8c_c5441f2db41b.slice/crio-05740d0e3bd149ecb73057bdeb6acef8335f7da3698e765450e82ee99c713fa1 WatchSource:0}: Error finding container 05740d0e3bd149ecb73057bdeb6acef8335f7da3698e765450e82ee99c713fa1: Status 404 returned error can't find the container with id 05740d0e3bd149ecb73057bdeb6acef8335f7da3698e765450e82ee99c713fa1 Oct 29 23:00:33.723278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.723253 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n"] Oct 29 23:00:33.838297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:33.838263 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" event={"ID":"f81e3ce7-b41b-4666-bb8c-c5441f2db41b","Type":"ContainerStarted","Data":"05740d0e3bd149ecb73057bdeb6acef8335f7da3698e765450e82ee99c713fa1"} Oct 29 23:00:34.849005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:34.848970 5367 generic.go:356] "Generic (PLEG): container finished" podID="f81e3ce7-b41b-4666-bb8c-c5441f2db41b" containerID="0fcbb20bb8cc9bd16fe354219385a1a9ea7b2bc003f382e29d6c04c4114c2821" exitCode=0 Oct 29 23:00:34.849430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:34.849045 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" event={"ID":"f81e3ce7-b41b-4666-bb8c-c5441f2db41b","Type":"ContainerDied","Data":"0fcbb20bb8cc9bd16fe354219385a1a9ea7b2bc003f382e29d6c04c4114c2821"} Oct 29 23:00:36.228316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.228285 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:36.302495 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.302438 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca\") pod \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " Oct 29 23:00:36.303295 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.303271 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca\") pod \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " Oct 29 23:00:36.303846 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.303828 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w\" (UniqueName: \"kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w\") pod \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\" (UID: \"f81e3ce7-b41b-4666-bb8c-c5441f2db41b\") " Oct 29 23:00:36.304438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.302994 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca" pod "f81e3ce7-b41b-4666-bb8c-c5441f2db41b" (UID: "f81e3ce7-b41b-4666-bb8c-c5441f2db41b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:36.305652 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.305627 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.311054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.311027 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca" pod "f81e3ce7-b41b-4666-bb8c-c5441f2db41b" (UID: "f81e3ce7-b41b-4666-bb8c-c5441f2db41b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:36.313185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.313158 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w" pod "f81e3ce7-b41b-4666-bb8c-c5441f2db41b" (UID: "f81e3ce7-b41b-4666-bb8c-c5441f2db41b"). InnerVolumeSpecName "kube-api-access-4fk7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:36.406760 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.406693 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.406760 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.406761 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fk7w\" (UniqueName: \"kubernetes.io/projected/f81e3ce7-b41b-4666-bb8c-c5441f2db41b-kube-api-access-4fk7w\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.922324 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.922293 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" Oct 29 23:00:36.922519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.922319 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n" event={"ID":"f81e3ce7-b41b-4666-bb8c-c5441f2db41b","Type":"ContainerDied","Data":"05740d0e3bd149ecb73057bdeb6acef8335f7da3698e765450e82ee99c713fa1"} Oct 29 23:00:36.922519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:36.922388 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05740d0e3bd149ecb73057bdeb6acef8335f7da3698e765450e82ee99c713fa1" Oct 29 23:00:37.748258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.748232 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 23:00:37.827241 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827207 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.827453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827295 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.827453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827376 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.827453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827438 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.827656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827516 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.827656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827586 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.827656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827610 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:37.827845 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827794 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:37.827944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827890 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:37.827944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.827820 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.828103 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.828086 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.828275 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.828253 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:37.828389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.828291 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs\") pod \"02720c0f-0336-4df3-8f0c-19853dcd6c59\" (UID: \"02720c0f-0336-4df3-8f0c-19853dcd6c59\") " Oct 29 23:00:37.828576 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.828554 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:37.830455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.829096 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.830455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.829123 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.830455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.829137 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.830455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.829149 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.830455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.829163 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02720c0f-0336-4df3-8f0c-19853dcd6c59-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.830835 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.830552 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "kube-api-access-wk2js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:37.831350 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.831321 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:37.831795 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.831747 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:37.845869 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.845844 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token" pod "02720c0f-0336-4df3-8f0c-19853dcd6c59" (UID: "02720c0f-0336-4df3-8f0c-19853dcd6c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:37.930250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.930224 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.930250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.930249 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.930440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.930258 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02720c0f-0336-4df3-8f0c-19853dcd6c59-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.930440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.930267 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2js\" (UniqueName: \"kubernetes.io/projected/02720c0f-0336-4df3-8f0c-19853dcd6c59-kube-api-access-wk2js\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:37.934223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.934196 5367 generic.go:356] "Generic (PLEG): container finished" podID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerID="98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f" exitCode=0 Oct 29 23:00:37.934346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.934258 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" Oct 29 23:00:37.934346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.934272 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" event={"ID":"02720c0f-0336-4df3-8f0c-19853dcd6c59","Type":"ContainerDied","Data":"98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f"} Oct 29 23:00:37.934346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.934328 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv" event={"ID":"02720c0f-0336-4df3-8f0c-19853dcd6c59","Type":"ContainerDied","Data":"00e2447b4639fa9393d5fdb3968598376f0518a350f2f6c43554f70d9ca8c78b"} Oct 29 23:00:37.934495 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.934348 5367 scope.go:122] "RemoveContainer" containerID="98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f" Oct 29 23:00:37.947407 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.947390 5367 scope.go:122] "RemoveContainer" containerID="fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c" Oct 29 23:00:37.958784 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.958764 5367 scope.go:122] "RemoveContainer" containerID="98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f" Oct 29 23:00:37.959049 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:37.959026 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f\": container with ID starting with 98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f not found: ID does not exist" containerID="98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f" Oct 29 23:00:37.959114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.959058 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f"} err="failed to get container status \"98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f\": rpc error: code = NotFound desc = could not find container \"98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f\": container with ID starting with 98a6b49097ca5bf5b26b689ec3fe482c8fb34a7592b0b7e935aa804b699f3f9f not found: ID does not exist" Oct 29 23:00:37.959114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.959076 5367 scope.go:122] "RemoveContainer" containerID="fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c" Oct 29 23:00:37.959332 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:37.959313 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c\": container with ID starting with fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c not found: ID does not exist" containerID="fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c" Oct 29 23:00:37.959397 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.959341 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c"} err="failed to get container status \"fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c\": rpc error: code = NotFound desc = could not find container \"fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c\": container with ID starting with fe6dd31263981d9c10dd64b8b2172ac7b69f32edf1c862015f959b6b23216d3c not found: ID does not exist" Oct 29 23:00:37.994182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:37.994131 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv"] Oct 29 23:00:38.000600 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:38.000577 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6f83c-openshift-default-9b6c88589-nx7fv"] Oct 29 23:00:38.520723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:38.502024 5367 scope.go:122] "RemoveContainer" containerID="7e9af4d7fd849c4d11d03e316fe5b271c3215f3fe645f824a577854bea3ba192" Oct 29 23:00:38.593403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:38.593371 5367 scope.go:122] "RemoveContainer" containerID="847f07b2782344c3813f4614016e09e75c25a0ccc8f87cc20e2d7de0ec2bddaa" Oct 29 23:00:39.227475 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:39.227441 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" path="/var/lib/kubelet/pods/02720c0f-0336-4df3-8f0c-19853dcd6c59/volumes" Oct 29 23:00:42.027331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:42.027286 5367 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:42.179802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:42.179750 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:42.539826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:42.539781 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:47.650535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.650444 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w"] Oct 29 23:00:47.651193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651178 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:47.651242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651194 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:47.651242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651211 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:47.651242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651216 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:47.651242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651227 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:47.651242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651232 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:47.651242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651240 5367 cpu_manager.go:414] "containerMap: removing container" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651245 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651251 5367 cpu_manager.go:414] "containerMap: removing container" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651256 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651263 5367 cpu_manager.go:414] "containerMap: removing container" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651267 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651280 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f81e3ce7-b41b-4666-bb8c-c5441f2db41b" containerName="revoke-keys" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651285 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81e3ce7-b41b-4666-bb8c-c5441f2db41b" containerName="revoke-keys" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651295 5367 cpu_manager.go:414] "containerMap: removing container" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651300 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651310 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651315 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7531bdb4-381a-4039-bcea-5ed7379a8546" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651324 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651328 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651337 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651342 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3ae329-788f-4719-b10e-146467086b36" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651351 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651356 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651364 5367 cpu_manager.go:414] "containerMap: removing container" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651369 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651377 5367 cpu_manager.go:414] "containerMap: removing container" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651381 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81935cce-192e-4569-afe7-b61fe0a9a419" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651394 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651398 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdf2542-f96c-4c37-88e7-fdb0a7778b43" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651406 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651410 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64ffbae-aa0d-4478-8a93-07c0094d7c63" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651423 5367 cpu_manager.go:414] "containerMap: removing container" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:47.651436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651427 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af08130d-e1d7-41eb-abe7-f626b698231f" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651518 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee465e5-6f19-4d52-9e85-00e40b15bd06" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651530 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ba151d-e166-4c58-8d4e-6e507a0986e1" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651537 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdec272-136f-4d21-a496-fe04aede5621" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651543 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651550 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba7d901-6282-41be-ad0f-fd74055bad81" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651555 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="def8b4aa-4cbf-43d5-ac35-43a87301a566" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651560 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81e3ce7-b41b-4666-bb8c-c5441f2db41b" containerName="revoke-keys" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651566 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d119d843-8ea0-444b-9590-614da9a4f435" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651571 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f056935-3821-4d4a-8fe6-7a9e41a9c6eb" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651577 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="02720c0f-0336-4df3-8f0c-19853dcd6c59" containerName="istio-proxy" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651583 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="722b2177-f08d-4203-b365-310ef1c795a5" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651590 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8003be9d-b7c7-4b0a-b988-02d13f22102b" containerName="cleanup" Oct 29 23:00:47.652297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.651597 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a80962-cbc0-4103-a7ef-387f88af4fdb" containerName="cleanup" Oct 29 23:00:47.680862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.680827 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:47.687209 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.687178 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w"] Oct 29 23:00:47.697316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.697288 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ee1fc\"" Oct 29 23:00:47.703468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.703437 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wm7\" (UniqueName: \"kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7\") pod \"payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:47.703590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.703522 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:47.732018 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.731985 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc"] Oct 29 23:00:47.737910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.737888 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:47.775722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.767013 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ee1fc-dockercfg-gmx47\"" Oct 29 23:00:47.820726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.808819 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:47.820726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.808939 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:47.820726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.809180 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8l99\" (UniqueName: \"kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:47.820726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.817833 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wm7\" (UniqueName: \"kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7\") pod \"payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:47.820726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:47.819400 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:48.140722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.136752 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:48.140722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.137005 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8l99\" (UniqueName: \"kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:48.141163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.141131 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:48.143826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.143804 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc"] Oct 29 23:00:48.149080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.147791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wm7\" (UniqueName: \"kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7\") pod \"payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:48.178401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.178370 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8l99\" (UniqueName: \"kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:48.300662 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.300630 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:48.351910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.351877 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:48.617738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.617693 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w"] Oct 29 23:00:48.617892 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:48.617866 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f27ecfa_32cf_4baa_a051_038e9ea1ceaf.slice/crio-6b0705e4759e4836040fd25c7643a69ebbc79cc26eb03821d3ecf8a47d5035d6 WatchSource:0}: Error finding container 6b0705e4759e4836040fd25c7643a69ebbc79cc26eb03821d3ecf8a47d5035d6: Status 404 returned error can't find the container with id 6b0705e4759e4836040fd25c7643a69ebbc79cc26eb03821d3ecf8a47d5035d6 Oct 29 23:00:48.683457 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:48.683421 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff9a057_03e0_4b3a_837f_3fc98618e7eb.slice/crio-b4e6b48287b996045b2b9e50545460a8f4e410af14faebfc6dc1c157b8b4eabb WatchSource:0}: Error finding container b4e6b48287b996045b2b9e50545460a8f4e410af14faebfc6dc1c157b8b4eabb: Status 404 returned error can't find the container with id b4e6b48287b996045b2b9e50545460a8f4e410af14faebfc6dc1c157b8b4eabb Oct 29 23:00:48.683910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:48.683795 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc"] Oct 29 23:00:49.272067 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.272026 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" event={"ID":"dff9a057-03e0-4b3a-837f-3fc98618e7eb","Type":"ContainerStarted","Data":"e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e"} Oct 29 23:00:49.272273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.272086 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" event={"ID":"dff9a057-03e0-4b3a-837f-3fc98618e7eb","Type":"ContainerStarted","Data":"b4e6b48287b996045b2b9e50545460a8f4e410af14faebfc6dc1c157b8b4eabb"} Oct 29 23:00:49.272273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.272251 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:00:49.274015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.273990 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" event={"ID":"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf","Type":"ContainerStarted","Data":"287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e"} Oct 29 23:00:49.274159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.274039 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:00:49.274159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.274064 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" event={"ID":"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf","Type":"ContainerStarted","Data":"6b0705e4759e4836040fd25c7643a69ebbc79cc26eb03821d3ecf8a47d5035d6"} Oct 29 23:00:49.323682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.323626 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" podStartSLOduration=2.323612946 podStartE2EDuration="2.323612946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:49.318321649 +0000 UTC m=+2716.212396571" watchObservedRunningTime="2026-10-29 23:00:49.323612946 +0000 UTC m=+2716.217687867" Oct 29 23:00:49.366363 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:49.366313 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" podStartSLOduration=2.366299673 podStartE2EDuration="2.366299673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:49.363661183 +0000 UTC m=+2716.257736107" watchObservedRunningTime="2026-10-29 23:00:49.366299673 +0000 UTC m=+2716.260374591" Oct 29 23:00:52.341806 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:00:52.341769 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:00:53.311636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:53.311602 5367 generic.go:356] "Generic (PLEG): container finished" podID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerID="d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543" exitCode=0 Oct 29 23:00:53.311868 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:53.311678 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" event={"ID":"43abfea5-a446-4ccf-be1b-b0dbcc19a091","Type":"ContainerDied","Data":"d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543"} Oct 29 23:00:53.312237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:53.312217 5367 scope.go:122] "RemoveContainer" containerID="d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543" Oct 29 23:00:53.314286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:53.314258 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:53.314365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:53.314331 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:53.314416 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:53.314366 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:54.377498 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.377441 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" event={"ID":"43abfea5-a446-4ccf-be1b-b0dbcc19a091","Type":"ContainerStarted","Data":"555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc"} Oct 29 23:00:54.585650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.585610 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg"] Oct 29 23:00:54.589414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.589388 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.635020 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.634963 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg"] Oct 29 23:00:54.638361 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.638343 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:54.638683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.638667 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:00:54.641357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.641337 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:00:54.681880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.681836 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.682166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.682144 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.687059 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.687020 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl5j\" (UniqueName: \"kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.738830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.738800 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:00:54.950802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.943241 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl5j\" (UniqueName: \"kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.951304 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.951269 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.951527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.951507 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.952105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.952080 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.981760 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.954833 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:54.994247 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:54.994214 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl5j\" (UniqueName: \"kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:55.076724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:55.071173 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 23:00:55.076724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:55.071307 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 23:00:55.076724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:55.071350 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 23:00:55.204362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:55.204285 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:55.799219 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:00:55.799179 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1df3df_06e4_46e1_b36e_4b963d8149e3.slice/crio-4e55f41942c0c1d2939c9968f5fdddd4f0fc4d1bd9d4899a6c759d6ad8b28ad2 WatchSource:0}: Error finding container 4e55f41942c0c1d2939c9968f5fdddd4f0fc4d1bd9d4899a6c759d6ad8b28ad2: Status 404 returned error can't find the container with id 4e55f41942c0c1d2939c9968f5fdddd4f0fc4d1bd9d4899a6c759d6ad8b28ad2 Oct 29 23:00:55.799768 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:55.799738 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg"] Oct 29 23:00:56.071722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:56.071136 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 23:00:56.071722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:56.071203 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 23:00:56.518679 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:56.518584 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" event={"ID":"2f1df3df-06e4-46e1-b36e-4b963d8149e3","Type":"ContainerStarted","Data":"de7908022c8478194a2a81e26a9ed4cc0d5383d658ada8f8621567d201702590"} Oct 29 23:00:56.518679 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:56.518643 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" event={"ID":"2f1df3df-06e4-46e1-b36e-4b963d8149e3","Type":"ContainerStarted","Data":"4e55f41942c0c1d2939c9968f5fdddd4f0fc4d1bd9d4899a6c759d6ad8b28ad2"} Oct 29 23:00:56.583194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:56.583138 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" podStartSLOduration=2.58312475 podStartE2EDuration="2.58312475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:56.58091206 +0000 UTC m=+2723.474986994" watchObservedRunningTime="2026-10-29 23:00:56.58312475 +0000 UTC m=+2723.477199669" Oct 29 23:00:57.086949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:57.086889 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 23:00:57.087532 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:57.087498 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 23:00:57.529208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:57.529174 5367 generic.go:356] "Generic (PLEG): container finished" podID="2f1df3df-06e4-46e1-b36e-4b963d8149e3" containerID="de7908022c8478194a2a81e26a9ed4cc0d5383d658ada8f8621567d201702590" exitCode=0 Oct 29 23:00:57.529419 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:57.529259 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" event={"ID":"2f1df3df-06e4-46e1-b36e-4b963d8149e3","Type":"ContainerDied","Data":"de7908022c8478194a2a81e26a9ed4cc0d5383d658ada8f8621567d201702590"} Oct 29 23:00:58.068231 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.068186 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 23:00:58.069313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.069277 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 23:00:58.825442 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.825410 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:58.910487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.910450 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca\") pod \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " Oct 29 23:00:58.910671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.910649 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j\" (UniqueName: \"kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j\") pod \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " Oct 29 23:00:58.910817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.910799 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca\") pod \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\" (UID: \"2f1df3df-06e4-46e1-b36e-4b963d8149e3\") " Oct 29 23:00:58.910891 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.910863 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca" pod "2f1df3df-06e4-46e1-b36e-4b963d8149e3" (UID: "2f1df3df-06e4-46e1-b36e-4b963d8149e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:58.911656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.911627 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:58.912815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.912775 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca" pod "2f1df3df-06e4-46e1-b36e-4b963d8149e3" (UID: "2f1df3df-06e4-46e1-b36e-4b963d8149e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:58.912902 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:58.912888 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j" pod "2f1df3df-06e4-46e1-b36e-4b963d8149e3" (UID: "2f1df3df-06e4-46e1-b36e-4b963d8149e3"). InnerVolumeSpecName "kube-api-access-qpl5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:59.013081 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.013032 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1df3df-06e4-46e1-b36e-4b963d8149e3-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:59.013081 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.013070 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpl5j\" (UniqueName: \"kubernetes.io/projected/2f1df3df-06e4-46e1-b36e-4b963d8149e3-kube-api-access-qpl5j\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:00:59.077723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.076820 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 23:00:59.077723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.076900 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 23:00:59.553215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.553123 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" Oct 29 23:00:59.553215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.553153 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg" event={"ID":"2f1df3df-06e4-46e1-b36e-4b963d8149e3","Type":"ContainerDied","Data":"4e55f41942c0c1d2939c9968f5fdddd4f0fc4d1bd9d4899a6c759d6ad8b28ad2"} Oct 29 23:00:59.553215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:00:59.553202 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e55f41942c0c1d2939c9968f5fdddd4f0fc4d1bd9d4899a6c759d6ad8b28ad2" Oct 29 23:01:00.146598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.146565 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 23:01:00.147329 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.147300 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 23:01:00.148110 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.148089 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 23:01:00.288229 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.288197 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:01:00.288423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.288277 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:01:00.647100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.647061 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd"] Oct 29 23:01:00.647873 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.647852 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2f1df3df-06e4-46e1-b36e-4b963d8149e3" containerName="revoke-keys" Oct 29 23:01:00.647873 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.647873 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1df3df-06e4-46e1-b36e-4b963d8149e3" containerName="revoke-keys" Oct 29 23:01:00.648116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.648101 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1df3df-06e4-46e1-b36e-4b963d8149e3" containerName="revoke-keys" Oct 29 23:01:00.650935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.650917 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.669619 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.669587 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3e2b2c-dockercfg-zf4fn\"" Oct 29 23:01:00.670445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.670417 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3e2b2c\"" Oct 29 23:01:00.676809 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.676782 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd"] Oct 29 23:01:00.718838 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.718806 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7"] Oct 29 23:01:00.722532 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.722508 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.765733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.757896 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpts8\" (UniqueName: \"kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8\") pod \"payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.765733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.758018 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.765733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.758213 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pd7z\" (UniqueName: \"kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.765733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.758352 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.849722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.834951 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7"] Oct 29 23:01:00.860172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.860127 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.860573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.860552 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpts8\" (UniqueName: \"kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8\") pod \"payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.860794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.860772 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.861127 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.861102 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pd7z\" (UniqueName: \"kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.882726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.862579 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.883073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.883039 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.943521 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.943434 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pd7z\" (UniqueName: \"kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:00.969251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.969218 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpts8\" (UniqueName: \"kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8\") pod \"payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:00.989566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:00.989530 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:01.118228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.118199 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:01.415899 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.415855 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd"] Oct 29 23:01:01.454439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.454413 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7"] Oct 29 23:01:01.457091 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:01.457061 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb144f021_128f_4b08_83b8_46fdb7fee0bb.slice/crio-8f6dce0f9fc7dc1adb37cad1114faa2888f39ffc4bd3101e96fff121714cb22c WatchSource:0}: Error finding container 8f6dce0f9fc7dc1adb37cad1114faa2888f39ffc4bd3101e96fff121714cb22c: Status 404 returned error can't find the container with id 8f6dce0f9fc7dc1adb37cad1114faa2888f39ffc4bd3101e96fff121714cb22c Oct 29 23:01:01.575773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.575734 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" event={"ID":"b144f021-128f-4b08-83b8-46fdb7fee0bb","Type":"ContainerStarted","Data":"257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978"} Oct 29 23:01:01.575919 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.575795 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" event={"ID":"b144f021-128f-4b08-83b8-46fdb7fee0bb","Type":"ContainerStarted","Data":"8f6dce0f9fc7dc1adb37cad1114faa2888f39ffc4bd3101e96fff121714cb22c"} Oct 29 23:01:01.575919 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.575904 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:01.577758 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.577728 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" event={"ID":"6a9c8e10-9a9e-497e-85fc-e0b241c25f60","Type":"ContainerStarted","Data":"0e6325ced3cb356bab8674d58d452a6a0c8ba6e9d5c9137d056261693a704842"} Oct 29 23:01:01.577884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.577856 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:01.637419 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.637361 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" podStartSLOduration=1.6373450269999998 podStartE2EDuration="1.637345027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:01.633317236 +0000 UTC m=+2728.527392155" watchObservedRunningTime="2026-10-29 23:01:01.637345027 +0000 UTC m=+2728.531419994" Oct 29 23:01:01.684177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:01.684104 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" podStartSLOduration=1.684083534 podStartE2EDuration="1.684083534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:01.680284418 +0000 UTC m=+2728.574359341" watchObservedRunningTime="2026-10-29 23:01:01.684083534 +0000 UTC m=+2728.578158461" Oct 29 23:01:02.589034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:02.588992 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" event={"ID":"6a9c8e10-9a9e-497e-85fc-e0b241c25f60","Type":"ContainerStarted","Data":"751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104"} Oct 29 23:01:05.489945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:05.489893 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf"] Oct 29 23:01:05.494822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:05.494788 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-40cf9a-nbgqf"] Oct 29 23:01:07.226221 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:07.226190 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537fe2b0-7515-4242-bb6f-fedd9d53f2a3" path="/var/lib/kubelet/pods/537fe2b0-7515-4242-bb6f-fedd9d53f2a3/volumes" Oct 29 23:01:12.618080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:12.617979 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:01:12.880498 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:12.880417 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg"] Oct 29 23:01:12.892875 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:12.892843 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-329f7b-rgtfg"] Oct 29 23:01:13.225709 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.225617 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1df3df-06e4-46e1-b36e-4b963d8149e3" path="/var/lib/kubelet/pods/2f1df3df-06e4-46e1-b36e-4b963d8149e3/volumes" Oct 29 23:01:13.691940 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.691909 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:01:13.749179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.749135 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj"] Oct 29 23:01:13.769857 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.769818 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.828037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.828001 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj"] Oct 29 23:01:13.893152 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.893109 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.894738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.893186 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq94r\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.894738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.893294 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.894738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.893390 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.895373 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.895341 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.895509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.895495 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.895645 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.895578 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.895759 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.895671 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.895759 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.895728 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.998645 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.998555 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.998645 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.998632 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.998693 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.998777 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.998864 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.998902 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq94r\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999001 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999093 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999113 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999142 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999170 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999207 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999430 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:13.999840 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:13.999818 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:14.001253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.001226 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:14.001373 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.001309 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:14.029227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.029196 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq94r\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:14.034965 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.034930 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:14.097438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.097403 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:14.105852 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.105811 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:01:14.106007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.105883 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:01:14.491586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.491550 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj"] Oct 29 23:01:14.509706 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:14.496515 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d3306f_e6fa_4dba_97f5_c1b968c7ac19.slice/crio-42f3f5f8208a915c7757d6367b16507b19a1085f369b4f4704ada1b58b4a3c8f WatchSource:0}: Error finding container 42f3f5f8208a915c7757d6367b16507b19a1085f369b4f4704ada1b58b4a3c8f: Status 404 returned error can't find the container with id 42f3f5f8208a915c7757d6367b16507b19a1085f369b4f4704ada1b58b4a3c8f Oct 29 23:01:14.509706 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.499501 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:14.509706 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.499566 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:14.509706 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.499596 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:14.866439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.865550 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" event={"ID":"62d3306f-e6fa-4dba-97f5-c1b968c7ac19","Type":"ContainerStarted","Data":"2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867"} Oct 29 23:01:14.866439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.865620 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" event={"ID":"62d3306f-e6fa-4dba-97f5-c1b968c7ac19","Type":"ContainerStarted","Data":"42f3f5f8208a915c7757d6367b16507b19a1085f369b4f4704ada1b58b4a3c8f"} Oct 29 23:01:14.929362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:14.929305 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podStartSLOduration=1.929285315 podStartE2EDuration="1.929285315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:14.92685268 +0000 UTC m=+2741.820927603" watchObservedRunningTime="2026-10-29 23:01:14.929285315 +0000 UTC m=+2741.823360240" Oct 29 23:01:15.097525 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.097487 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:15.099092 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.099068 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:15.099190 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.099116 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:15.276342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.276261 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6"] Oct 29 23:01:15.279905 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.279887 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.320988 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.320949 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321004 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321169 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwhk\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321383 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321495 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321468 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321644 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321627 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321670 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321770 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321741 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.321830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.321797 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.324687 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.324659 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6"] Oct 29 23:01:15.423491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423457 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.423673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423560 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.423673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423582 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.423673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423616 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.423673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423647 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.423673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423669 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423826 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423913 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwhk\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423951 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.423993 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.424052 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.424063 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.424092 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.424498 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.424477 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.425974 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.425957 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.426513 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.426490 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.457030 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.457002 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwhk\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.458921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.458899 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.593181 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.593146 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:15.890408 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.890319 5367 generic.go:356] "Generic (PLEG): container finished" podID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerID="69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8" exitCode=0 Oct 29 23:01:15.890873 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.890507 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" event={"ID":"466bb494-cc30-4ab1-b12d-b6157e64f152","Type":"ContainerDied","Data":"69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8"} Oct 29 23:01:15.891362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.891344 5367 scope.go:122] "RemoveContainer" containerID="69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8" Oct 29 23:01:15.893573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.893543 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:15.893670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.893612 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:15.893670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.893643 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:15.938026 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:15.938000 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccb8c56_ea8b_4438_a080_b27780ad23ac.slice/crio-0b9b0802cf50db1c82e8b374cf2a5eefb8376fbaa815e0eb360664bd7dbc3720 WatchSource:0}: Error finding container 0b9b0802cf50db1c82e8b374cf2a5eefb8376fbaa815e0eb360664bd7dbc3720: Status 404 returned error can't find the container with id 0b9b0802cf50db1c82e8b374cf2a5eefb8376fbaa815e0eb360664bd7dbc3720 Oct 29 23:01:15.939381 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.939353 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6"] Oct 29 23:01:15.941363 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.941327 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:15.941464 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.941417 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:15.941533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:15.941462 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:01:16.098022 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.097981 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.098730 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.098264 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:16.902635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.902596 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" event={"ID":"466bb494-cc30-4ab1-b12d-b6157e64f152","Type":"ContainerStarted","Data":"9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099"} Oct 29 23:01:16.904410 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.904373 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" event={"ID":"4ccb8c56-ea8b-4438-a080-b27780ad23ac","Type":"ContainerStarted","Data":"6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d"} Oct 29 23:01:16.904555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.904424 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" event={"ID":"4ccb8c56-ea8b-4438-a080-b27780ad23ac","Type":"ContainerStarted","Data":"0b9b0802cf50db1c82e8b374cf2a5eefb8376fbaa815e0eb360664bd7dbc3720"} Oct 29 23:01:16.973586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.972812 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 23:01:16.973586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.972972 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.973586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:16.973021 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:17.044000 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.043951 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podStartSLOduration=2.0439378 podStartE2EDuration="2.0439378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:17.040200049 +0000 UTC m=+2743.934274969" watchObservedRunningTime="2026-10-29 23:01:17.0439378 +0000 UTC m=+2743.938012721" Oct 29 23:01:17.097939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.097903 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:17.098122 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.097954 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:17.593973 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.593936 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:17.595664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.595640 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 23:01:17.595780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.595691 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 23:01:17.994788 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.991469 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:17.994788 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:17.991541 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:18.041526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.041464 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6"] Oct 29 23:01:18.042011 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.041980 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" containerID="cri-o://83c72c5c8a35ab076d6ca3919b5f26d62050923ba8a09fabe9f051d844ef62f3" gracePeriod=30 Oct 29 23:01:18.098282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.098236 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:18.102827 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.102784 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:18.608228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.608185 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 23:01:18.608545 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.608514 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 23:01:18.964786 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.961879 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:18.964786 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:18.961954 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:19.107723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:19.107132 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:19.107723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:19.107195 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:19.605723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:19.604283 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 23:01:19.605723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:19.604350 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 23:01:20.021783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.020851 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.021783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.020933 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:20.172129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.172094 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.262732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.188744 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:20.614727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.597330 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.615070 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.615032 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 23:01:20.937790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.934316 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:01:20.982723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.982178 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.982723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:20.982357 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:21.151260 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.121559 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:21.151260 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.121623 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:21.233130 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.233020 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:01:21.613761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.613373 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 23:01:21.613761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.613439 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 23:01:21.999783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.986244 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:21.999783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:21.986302 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:22.108725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:22.103085 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Oct 29 23:01:22.108725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:22.103154 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Oct 29 23:01:22.470938 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:22.470884 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:01:22.609755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:22.594749 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 23:01:22.609755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:22.594806 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 23:01:23.021723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.009225 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:23.021723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.009301 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:23.037076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.034785 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6"] Oct 29 23:01:23.037076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.035086 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" containerID="cri-o://6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d" gracePeriod=30 Oct 29 23:01:23.115721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.107834 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj"] Oct 29 23:01:23.115721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.108100 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" containerID="cri-o://e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1" gracePeriod=30 Oct 29 23:01:23.386874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.383085 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:23.386874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.384757 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:23.415829 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.415124 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:01:23.541066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.540987 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn"] Oct 29 23:01:23.546196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.546172 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.548217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.548197 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn"] Oct 29 23:01:23.570721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.567954 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:01:23.611163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.585404 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:01:23.611163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.590467 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:01:23.611163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.590547 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:01:23.713770 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.713740 5367 generic.go:356] "Generic (PLEG): container finished" podID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerID="83c72c5c8a35ab076d6ca3919b5f26d62050923ba8a09fabe9f051d844ef62f3" exitCode=0 Oct 29 23:01:23.714808 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.714780 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" event={"ID":"c15a9f08-4c97-481b-88bd-7f48c6f0e488","Type":"ContainerDied","Data":"83c72c5c8a35ab076d6ca3919b5f26d62050923ba8a09fabe9f051d844ef62f3"} Oct 29 23:01:23.740655 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.740618 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.741554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.741092 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwsjd\" (UniqueName: \"kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.743270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.743243 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.845402 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.845339 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.845861 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.845829 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.846051 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.846017 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwsjd\" (UniqueName: \"kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.847383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.846251 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.853279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.853249 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.909727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.896472 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwsjd\" (UniqueName: \"kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.959723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.954483 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:23.984725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.960206 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 29 23:01:23.984725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:23.960256 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 29 23:01:24.024423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.024384 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 23:01:24.048262 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048227 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048309 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048374 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048413 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048487 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048510 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048534 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048560 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048618 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token\") pod \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\" (UID: \"c15a9f08-4c97-481b-88bd-7f48c6f0e488\") " Oct 29 23:01:24.048982 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048738 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:24.049423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.049395 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.050826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048882 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:24.050976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048899 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:24.050976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.048908 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:24.050976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.049199 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:24.051177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.051110 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:24.051330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.051311 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "kube-api-access-49h72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:24.051585 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.051561 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:24.052237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.052206 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token" pod "c15a9f08-4c97-481b-88bd-7f48c6f0e488" (UID: "c15a9f08-4c97-481b-88bd-7f48c6f0e488"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161236 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161272 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161285 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161300 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161313 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161326 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/c15a9f08-4c97-481b-88bd-7f48c6f0e488-kube-api-access-49h72\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161339 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.161519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.161352 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15a9f08-4c97-481b-88bd-7f48c6f0e488-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.723692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.723611 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" Oct 29 23:01:24.723692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.723637 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6" event={"ID":"c15a9f08-4c97-481b-88bd-7f48c6f0e488","Type":"ContainerDied","Data":"b83e29310b5d8e900ec1eaae1a82ab98e1e01516f9297b4cb56dd8b2ab545514"} Oct 29 23:01:24.724255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.723760 5367 scope.go:122] "RemoveContainer" containerID="83c72c5c8a35ab076d6ca3919b5f26d62050923ba8a09fabe9f051d844ef62f3" Oct 29 23:01:24.789388 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.789354 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6"] Oct 29 23:01:24.801592 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.801566 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-tzxh6"] Oct 29 23:01:24.888675 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:24.888648 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cf1159_97a4_4021_87b8_554cfe5e1519.slice/crio-31c57d47d6991a0cb8acb271519c12f80c8ab1aa2512a6900c7bc50443d653df WatchSource:0}: Error finding container 31c57d47d6991a0cb8acb271519c12f80c8ab1aa2512a6900c7bc50443d653df: Status 404 returned error can't find the container with id 31c57d47d6991a0cb8acb271519c12f80c8ab1aa2512a6900c7bc50443d653df Oct 29 23:01:24.891898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:24.891869 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn"] Oct 29 23:01:25.054410 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.054371 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n"] Oct 29 23:01:25.062296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.062267 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcb810-6r95n"] Oct 29 23:01:25.118372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.118340 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 23:01:25.119306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.119183 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 23:01:25.120223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.120202 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 23:01:25.232749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.228020 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" path="/var/lib/kubelet/pods/c15a9f08-4c97-481b-88bd-7f48c6f0e488/volumes" Oct 29 23:01:25.232749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.228855 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81e3ce7-b41b-4666-bb8c-c5441f2db41b" path="/var/lib/kubelet/pods/f81e3ce7-b41b-4666-bb8c-c5441f2db41b/volumes" Oct 29 23:01:25.735369 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.735333 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" event={"ID":"75cf1159-97a4-4021-87b8-554cfe5e1519","Type":"ContainerStarted","Data":"5cc46314cb4fffd0ea606657276b007fa57f54987ad89548d56f66a96dab904f"} Oct 29 23:01:25.735892 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.735396 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" event={"ID":"75cf1159-97a4-4021-87b8-554cfe5e1519","Type":"ContainerStarted","Data":"31c57d47d6991a0cb8acb271519c12f80c8ab1aa2512a6900c7bc50443d653df"} Oct 29 23:01:25.791872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:25.791826 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" podStartSLOduration=2.79181254 podStartE2EDuration="2.79181254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:25.788784713 +0000 UTC m=+2752.682859635" watchObservedRunningTime="2026-10-29 23:01:25.79181254 +0000 UTC m=+2752.685887486" Oct 29 23:01:26.583853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:26.583819 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg"] Oct 29 23:01:26.584053 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:26.584030 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" containerID="cri-o://832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1" gracePeriod=30 Oct 29 23:01:26.838384 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:26.838292 5367 generic.go:356] "Generic (PLEG): container finished" podID="75cf1159-97a4-4021-87b8-554cfe5e1519" containerID="5cc46314cb4fffd0ea606657276b007fa57f54987ad89548d56f66a96dab904f" exitCode=0 Oct 29 23:01:26.839447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:26.839418 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" event={"ID":"75cf1159-97a4-4021-87b8-554cfe5e1519","Type":"ContainerDied","Data":"5cc46314cb4fffd0ea606657276b007fa57f54987ad89548d56f66a96dab904f"} Oct 29 23:01:27.005130 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.005091 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78"] Oct 29 23:01:27.006956 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.006930 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" Oct 29 23:01:27.014157 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.014130 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" Oct 29 23:01:27.014563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.014542 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15a9f08-4c97-481b-88bd-7f48c6f0e488" containerName="istio-proxy" Oct 29 23:01:27.018260 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.018238 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.037675 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.037602 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 23:01:27.038635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.038599 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 23:01:27.038806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.038662 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2d286f-serving-cert\"" Oct 29 23:01:27.040867 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.040844 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78"] Oct 29 23:01:27.096735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.096647 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.096907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.096890 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5k4t\" (UniqueName: \"kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.096969 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.096953 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.097061 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.097038 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.198293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.198262 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5k4t\" (UniqueName: \"kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.198449 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.198331 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.198491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.198475 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.198681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.198660 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.199268 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.199243 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.200747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.200728 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.200964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.200942 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.253542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.250680 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5k4t\" (UniqueName: \"kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t\") pod \"maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.337270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.337235 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:27.656555 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:27.656530 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c75367_0bea_44db_a48f_424299e6e3e5.slice/crio-c3d0114b07c2cde05ffd66fbfdfae833572b2c91f67445c565b377036d1ad93f WatchSource:0}: Error finding container c3d0114b07c2cde05ffd66fbfdfae833572b2c91f67445c565b377036d1ad93f: Status 404 returned error can't find the container with id c3d0114b07c2cde05ffd66fbfdfae833572b2c91f67445c565b377036d1ad93f Oct 29 23:01:27.657610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.657512 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78"] Oct 29 23:01:27.658572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.658554 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:01:27.850092 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:27.850056 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" event={"ID":"95c75367-0bea-44db-a48f-424299e6e3e5","Type":"ContainerStarted","Data":"c3d0114b07c2cde05ffd66fbfdfae833572b2c91f67445c565b377036d1ad93f"} Oct 29 23:01:28.024346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.024321 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:28.138781 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.116786 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca\") pod \"75cf1159-97a4-4021-87b8-554cfe5e1519\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " Oct 29 23:01:28.138781 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.117074 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd\" (UniqueName: \"kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd\") pod \"75cf1159-97a4-4021-87b8-554cfe5e1519\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " Oct 29 23:01:28.154368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.142237 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca\") pod \"75cf1159-97a4-4021-87b8-554cfe5e1519\" (UID: \"75cf1159-97a4-4021-87b8-554cfe5e1519\") " Oct 29 23:01:28.158661 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.157216 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd" pod "75cf1159-97a4-4021-87b8-554cfe5e1519" (UID: "75cf1159-97a4-4021-87b8-554cfe5e1519"). InnerVolumeSpecName "kube-api-access-pwsjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.159680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.159601 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca" pod "75cf1159-97a4-4021-87b8-554cfe5e1519" (UID: "75cf1159-97a4-4021-87b8-554cfe5e1519"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:28.162859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.162820 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca" pod "75cf1159-97a4-4021-87b8-554cfe5e1519" (UID: "75cf1159-97a4-4021-87b8-554cfe5e1519"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:28.244292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.244265 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.244292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.244291 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwsjd\" (UniqueName: \"kubernetes.io/projected/75cf1159-97a4-4021-87b8-554cfe5e1519-kube-api-access-pwsjd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.244475 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.244302 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75cf1159-97a4-4021-87b8-554cfe5e1519-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.537129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.537043 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 23:01:28.562079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562030 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562126 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562155 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562199 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562295 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562349 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562411 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562483 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.562672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.562614 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket\") pod \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\" (UID: \"2f12f3fb-0d53-4b06-a490-c93a9510aa91\") " Oct 29 23:01:28.564945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.564039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.564945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.564139 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.564945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.564233 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.564945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.564575 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:28.564945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.564574 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.569189 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.567806 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.570834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.570747 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "kube-api-access-hl2nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.571040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.571021 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:28.571889 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.571854 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token" pod "2f12f3fb-0d53-4b06-a490-c93a9510aa91" (UID: "2f12f3fb-0d53-4b06-a490-c93a9510aa91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.665384 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665349 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665384 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665380 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665384 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665389 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665397 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665405 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665413 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl2nw\" (UniqueName: \"kubernetes.io/projected/2f12f3fb-0d53-4b06-a490-c93a9510aa91-kube-api-access-hl2nw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665423 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665431 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.665622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.665440 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f12f3fb-0d53-4b06-a490-c93a9510aa91-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.913833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.881908 5367 generic.go:356] "Generic (PLEG): container finished" podID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerID="e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1" exitCode=0 Oct 29 23:01:28.913833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.882020 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" event={"ID":"2f12f3fb-0d53-4b06-a490-c93a9510aa91","Type":"ContainerDied","Data":"e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1"} Oct 29 23:01:28.913833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.882078 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" event={"ID":"2f12f3fb-0d53-4b06-a490-c93a9510aa91","Type":"ContainerDied","Data":"be6ff00f4342e45b55e79d9ec4f046cf146e61467b3ab2be5a42fbe174db9548"} Oct 29 23:01:28.913833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.882111 5367 scope.go:122] "RemoveContainer" containerID="e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1" Oct 29 23:01:28.913833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:28.882262 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj" Oct 29 23:01:29.193269 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.161910 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" event={"ID":"75cf1159-97a4-4021-87b8-554cfe5e1519","Type":"ContainerDied","Data":"31c57d47d6991a0cb8acb271519c12f80c8ab1aa2512a6900c7bc50443d653df"} Oct 29 23:01:29.193269 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.161969 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c57d47d6991a0cb8acb271519c12f80c8ab1aa2512a6900c7bc50443d653df" Oct 29 23:01:29.193269 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.162045 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn" Oct 29 23:01:29.311609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.311558 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" event={"ID":"95c75367-0bea-44db-a48f-424299e6e3e5","Type":"ContainerStarted","Data":"db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0"} Oct 29 23:01:29.319451 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.319423 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj"] Oct 29 23:01:29.320160 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.320140 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:29.355204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.355166 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-wntrj"] Oct 29 23:01:29.383272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.383210 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" podStartSLOduration=3.025630476 podStartE2EDuration="3.383188338s" totalImagesPullingTime="357.557862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:29.372905617 +0000 UTC m=+2756.266980562" watchObservedRunningTime="2026-10-29 23:01:29.383188338 +0000 UTC m=+2756.277263260" Oct 29 23:01:29.454998 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.438986 5367 scope.go:122] "RemoveContainer" containerID="e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1" Oct 29 23:01:29.454998 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:29.439465 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1\": container with ID starting with e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1 not found: ID does not exist" containerID="e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1" Oct 29 23:01:29.454998 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:29.439505 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1"} err="failed to get container status \"e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1\": rpc error: code = NotFound desc = could not find container \"e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1\": container with ID starting with e20ce49c8fa7334a16eaaaa676dea9a4152a9f1a0164aeed378178981143a8a1 not found: ID does not exist" Oct 29 23:01:30.593936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:30.593905 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:30.890028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:30.890003 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:31.049428 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049390 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.049639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049471 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.049639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049506 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.049639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049606 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.049854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049674 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.049854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049783 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.049854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049799 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:31.050030 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.049868 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.050098 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050022 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:31.050098 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050036 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.050098 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050038 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:31.050276 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050116 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert\") pod \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\" (UID: \"4ccb8c56-ea8b-4438-a080-b27780ad23ac\") " Oct 29 23:01:31.050276 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050239 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:31.050590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050566 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:31.050818 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050801 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.050882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050821 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.050882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050830 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.050882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050839 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.050882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.050848 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.052259 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.052232 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:31.052380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.052299 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:31.052380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.052302 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:31.052380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.052301 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk" pod "4ccb8c56-ea8b-4438-a080-b27780ad23ac" (UID: "4ccb8c56-ea8b-4438-a080-b27780ad23ac"). InnerVolumeSpecName "kube-api-access-nkwhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:31.152131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.152055 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.152131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.152082 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwhk\" (UniqueName: \"kubernetes.io/projected/4ccb8c56-ea8b-4438-a080-b27780ad23ac-kube-api-access-nkwhk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.152131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.152092 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.152131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.152100 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb8c56-ea8b-4438-a080-b27780ad23ac-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.223991 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.223957 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" path="/var/lib/kubelet/pods/2f12f3fb-0d53-4b06-a490-c93a9510aa91/volumes" Oct 29 23:01:31.324168 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.324134 5367 generic.go:356] "Generic (PLEG): container finished" podID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerID="6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d" exitCode=0 Oct 29 23:01:31.324359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.324200 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" Oct 29 23:01:31.324359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.324219 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" event={"ID":"4ccb8c56-ea8b-4438-a080-b27780ad23ac","Type":"ContainerDied","Data":"6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d"} Oct 29 23:01:31.324359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.324281 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6" event={"ID":"4ccb8c56-ea8b-4438-a080-b27780ad23ac","Type":"ContainerDied","Data":"0b9b0802cf50db1c82e8b374cf2a5eefb8376fbaa815e0eb360664bd7dbc3720"} Oct 29 23:01:31.324359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.324304 5367 scope.go:122] "RemoveContainer" containerID="6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d" Oct 29 23:01:31.336179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.336161 5367 scope.go:122] "RemoveContainer" containerID="6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d" Oct 29 23:01:31.336462 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:31.336443 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d\": container with ID starting with 6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d not found: ID does not exist" containerID="6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d" Oct 29 23:01:31.336529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.336473 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d"} err="failed to get container status \"6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d\": rpc error: code = NotFound desc = could not find container \"6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d\": container with ID starting with 6e72493cd2ac799af135a5bbcffa5e8d56bc279469825aa44246c39e66c4202d not found: ID does not exist" Oct 29 23:01:31.397464 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.397428 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6"] Oct 29 23:01:31.402290 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.402226 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-pxhj6"] Oct 29 23:01:31.912947 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:31.912876 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.066584 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.066666 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.066752 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.066810 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.066856 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.066924 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.067007 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.067052 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.067107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.067100 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs\") pod \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\" (UID: \"03b4153f-06e8-43c1-9121-4cdbc45bb6b7\") " Oct 29 23:01:32.068481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.068089 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:32.070317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.070285 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:32.070443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.070302 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.071272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.070999 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:32.071272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.071243 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:32.071526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.071506 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:32.073909 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.073885 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns"] Oct 29 23:01:32.074412 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.074140 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" containerID="cri-o://76c303c76ad842c38f7ca1ed060220693bf67078c58aa9dc72f4d08d31a1a95e" gracePeriod=30 Oct 29 23:01:32.083146 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.083112 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "kube-api-access-qjbpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:32.090868 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.090745 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:32.090868 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.090829 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:32.092774 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.092748 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token" pod "03b4153f-06e8-43c1-9121-4cdbc45bb6b7" (UID: "03b4153f-06e8-43c1-9121-4cdbc45bb6b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172454 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172494 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172508 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjbpf\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-kube-api-access-qjbpf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172523 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172541 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172555 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172568 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.172671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.172580 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b4153f-06e8-43c1-9121-4cdbc45bb6b7-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:32.373112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.371239 5367 generic.go:356] "Generic (PLEG): container finished" podID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerID="832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1" exitCode=0 Oct 29 23:01:32.373112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.371331 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" event={"ID":"03b4153f-06e8-43c1-9121-4cdbc45bb6b7","Type":"ContainerDied","Data":"832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1"} Oct 29 23:01:32.373112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.371379 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" event={"ID":"03b4153f-06e8-43c1-9121-4cdbc45bb6b7","Type":"ContainerDied","Data":"6a737d8a3b0bf7eb2e7c2a7dd1720b16f39bc26099f50d52519a7b369798b929"} Oct 29 23:01:32.373112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.371408 5367 scope.go:122] "RemoveContainer" containerID="832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1" Oct 29 23:01:32.373112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.371551 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg" Oct 29 23:01:32.394590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.394563 5367 scope.go:122] "RemoveContainer" containerID="832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1" Oct 29 23:01:32.395011 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:32.394985 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1\": container with ID starting with 832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1 not found: ID does not exist" containerID="832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1" Oct 29 23:01:32.395140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.395022 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1"} err="failed to get container status \"832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1\": rpc error: code = NotFound desc = could not find container \"832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1\": container with ID starting with 832cfa241ea57b18f2284c26ab1d28acc72385bc2d8e1a26c947b3d840d7d3a1 not found: ID does not exist" Oct 29 23:01:32.466103 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.466017 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg"] Oct 29 23:01:32.470126 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.470102 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-hbvcg"] Oct 29 23:01:32.622440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.622405 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49"] Oct 29 23:01:32.623253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623230 5367 cpu_manager.go:414] "containerMap: removing container" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" Oct 29 23:01:32.623253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623253 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" Oct 29 23:01:32.623465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623280 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" Oct 29 23:01:32.623465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623286 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" Oct 29 23:01:32.623465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623308 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" Oct 29 23:01:32.623465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623316 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" Oct 29 23:01:32.623465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623342 5367 cpu_manager.go:414] "containerMap: removing container" podUID="75cf1159-97a4-4021-87b8-554cfe5e1519" containerName="revoke-keys" Oct 29 23:01:32.623465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623349 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf1159-97a4-4021-87b8-554cfe5e1519" containerName="revoke-keys" Oct 29 23:01:32.623804 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623519 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" containerName="istio-proxy" Oct 29 23:01:32.623804 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623536 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f12f3fb-0d53-4b06-a490-c93a9510aa91" containerName="istio-proxy" Oct 29 23:01:32.623804 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623547 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" containerName="istio-proxy" Oct 29 23:01:32.623804 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.623558 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf1159-97a4-4021-87b8-554cfe5e1519" containerName="revoke-keys" Oct 29 23:01:32.628325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.628305 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.648908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.648876 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:01:32.648908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.648899 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:01:32.654172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.654146 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49"] Oct 29 23:01:32.782515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.782424 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcpx\" (UniqueName: \"kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.782515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.782475 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.782747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.782614 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.883954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.883923 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcpx\" (UniqueName: \"kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.884174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.883968 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.884174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.884075 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.884801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.884783 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.886491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.886470 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.920461 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.920427 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcpx\" (UniqueName: \"kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:32.942988 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:32.942955 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:33.225225 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:33.225179 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b4153f-06e8-43c1-9121-4cdbc45bb6b7" path="/var/lib/kubelet/pods/03b4153f-06e8-43c1-9121-4cdbc45bb6b7/volumes" Oct 29 23:01:33.225922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:33.225904 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccb8c56-ea8b-4438-a080-b27780ad23ac" path="/var/lib/kubelet/pods/4ccb8c56-ea8b-4438-a080-b27780ad23ac/volumes" Oct 29 23:01:33.256037 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:33.256004 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffb3e9a_34f6_4159_b704_b9d0442397c9.slice/crio-477d95e33878631a610bebef803f9d23604c8aa016e0a366361b53f75035ddc8 WatchSource:0}: Error finding container 477d95e33878631a610bebef803f9d23604c8aa016e0a366361b53f75035ddc8: Status 404 returned error can't find the container with id 477d95e33878631a610bebef803f9d23604c8aa016e0a366361b53f75035ddc8 Oct 29 23:01:33.256790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:33.256766 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49"] Oct 29 23:01:33.383228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:33.383191 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" event={"ID":"affb3e9a-34f6-4159-b704-b9d0442397c9","Type":"ContainerStarted","Data":"477d95e33878631a610bebef803f9d23604c8aa016e0a366361b53f75035ddc8"} Oct 29 23:01:33.913547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:33.913449 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587"] Oct 29 23:01:33.949240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:33.949204 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn"] Oct 29 23:01:34.394454 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:34.394414 5367 generic.go:356] "Generic (PLEG): container finished" podID="affb3e9a-34f6-4159-b704-b9d0442397c9" containerID="c1d777dc103191b16ede5864b8049f85ef093941f7c30edf4e709e6c0f5cacc8" exitCode=0 Oct 29 23:01:34.394665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:34.394503 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" event={"ID":"affb3e9a-34f6-4159-b704-b9d0442397c9","Type":"ContainerDied","Data":"c1d777dc103191b16ede5864b8049f85ef093941f7c30edf4e709e6c0f5cacc8"} Oct 29 23:01:35.590969 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.590944 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:35.643502 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.643374 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca\") pod \"affb3e9a-34f6-4159-b704-b9d0442397c9\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " Oct 29 23:01:35.643826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.643514 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx\" (UniqueName: \"kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx\") pod \"affb3e9a-34f6-4159-b704-b9d0442397c9\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " Oct 29 23:01:35.643826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.643729 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca\") pod \"affb3e9a-34f6-4159-b704-b9d0442397c9\" (UID: \"affb3e9a-34f6-4159-b704-b9d0442397c9\") " Oct 29 23:01:35.644731 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.644684 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca" pod "affb3e9a-34f6-4159-b704-b9d0442397c9" (UID: "affb3e9a-34f6-4159-b704-b9d0442397c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:35.647234 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.647204 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca" pod "affb3e9a-34f6-4159-b704-b9d0442397c9" (UID: "affb3e9a-34f6-4159-b704-b9d0442397c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:35.649197 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.649173 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx" pod "affb3e9a-34f6-4159-b704-b9d0442397c9" (UID: "affb3e9a-34f6-4159-b704-b9d0442397c9"). InnerVolumeSpecName "kube-api-access-zlcpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:35.745557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.745523 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:35.745557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.745553 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affb3e9a-34f6-4159-b704-b9d0442397c9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:35.745557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.745563 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlcpx\" (UniqueName: \"kubernetes.io/projected/affb3e9a-34f6-4159-b704-b9d0442397c9-kube-api-access-zlcpx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:35.981378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.981286 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:35.981378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:35.981345 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:36.330488 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:36.330461 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:01:36.417755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:36.417715 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" event={"ID":"affb3e9a-34f6-4159-b704-b9d0442397c9","Type":"ContainerDied","Data":"477d95e33878631a610bebef803f9d23604c8aa016e0a366361b53f75035ddc8"} Oct 29 23:01:36.417755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:36.417752 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49" Oct 29 23:01:36.418026 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:36.417773 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477d95e33878631a610bebef803f9d23604c8aa016e0a366361b53f75035ddc8" Oct 29 23:01:37.454177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.454141 5367 generic.go:356] "Generic (PLEG): container finished" podID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerID="76c303c76ad842c38f7ca1ed060220693bf67078c58aa9dc72f4d08d31a1a95e" exitCode=0 Oct 29 23:01:37.454612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.454185 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" event={"ID":"933caef7-daeb-4332-97a6-5ea2aedfe708","Type":"ContainerDied","Data":"76c303c76ad842c38f7ca1ed060220693bf67078c58aa9dc72f4d08d31a1a95e"} Oct 29 23:01:37.454612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.454248 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" event={"ID":"933caef7-daeb-4332-97a6-5ea2aedfe708","Type":"ContainerDied","Data":"c84ae7a2063e0ca8393a26685c69a8462cf7f08956ef19735c7546b3b2774f5c"} Oct 29 23:01:37.454612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.454275 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84ae7a2063e0ca8393a26685c69a8462cf7f08956ef19735c7546b3b2774f5c" Oct 29 23:01:37.454612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.454293 5367 scope.go:122] "RemoveContainer" containerID="a8843d17970f09b33187e5ad49adef68078296495d75b14496fd54494052073f" Oct 29 23:01:37.475207 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.475172 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 23:01:37.610363 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610320 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.610583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610403 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.610583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610460 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.610791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610604 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.610791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610640 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.610791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610717 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:37.610791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610730 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:37.611065 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.610837 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:37.611141 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.611121 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.611203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.611158 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.611266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.611212 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.611325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.611293 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert\") pod \"933caef7-daeb-4332-97a6-5ea2aedfe708\" (UID: \"933caef7-daeb-4332-97a6-5ea2aedfe708\") " Oct 29 23:01:37.611439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.611413 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:37.611747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.611720 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:37.612155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.612136 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.612155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.612156 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933caef7-daeb-4332-97a6-5ea2aedfe708-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.612326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.612166 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.612326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.612174 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.612326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.612182 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.613067 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.613033 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "kube-api-access-7s7cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:37.613146 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.613118 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:37.613612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.613589 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:37.613833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.613815 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo" pod "933caef7-daeb-4332-97a6-5ea2aedfe708" (UID: "933caef7-daeb-4332-97a6-5ea2aedfe708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:37.713344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.713250 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s7cd\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-kube-api-access-7s7cd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.713344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.713284 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.713344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.713295 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:37.713344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:37.713305 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933caef7-daeb-4332-97a6-5ea2aedfe708-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:38.472639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.472563 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns" Oct 29 23:01:38.541512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.541478 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns"] Oct 29 23:01:38.553663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.553633 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcb810-openshift-default-56f69574df-l9lns"] Oct 29 23:01:38.857100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.857068 5367 scope.go:122] "RemoveContainer" containerID="15e7fdc34d19c5243a0fc1b195b5158dac9fb3d895d5471f387176d3af5f2936" Oct 29 23:01:38.868356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.868337 5367 scope.go:122] "RemoveContainer" containerID="59973597d66a79ed5234c1bb35c50b7e3931a9d18a4bff2c0b30b592991e377a" Oct 29 23:01:38.878977 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.878954 5367 scope.go:122] "RemoveContainer" containerID="27e4b2f966264ddf815e96fe7a15616a3900936bf995e7674e3b73c6cf19ff3e" Oct 29 23:01:38.889012 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.888992 5367 scope.go:122] "RemoveContainer" containerID="dbbed726f87d84ada14b904404d3b66fa139be4a296b0ae6f8b4903bc3e64185" Oct 29 23:01:38.899148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.899129 5367 scope.go:122] "RemoveContainer" containerID="6d0dee3408695d3ee6cac8a32b81407abc4517a4f637a346caaa3326b851f671" Oct 29 23:01:38.909283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.909252 5367 scope.go:122] "RemoveContainer" containerID="e5dbec99e5c272eda0389f9178110716682ef3e3b464878152c8f23e72d8ccea" Oct 29 23:01:38.914302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.914268 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerName="payload-processing" containerID="cri-o://642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407" gracePeriod=25 Oct 29 23:01:38.920268 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.920242 5367 scope.go:122] "RemoveContainer" containerID="dac0a3b76e1a9825bf3f03de3db2dbc16a78390c4b022ae469e7e183a73cb452" Oct 29 23:01:38.931340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.931316 5367 scope.go:122] "RemoveContainer" containerID="8511e1cb3d7e4d79ad8f5eccb6a54c3066b2919fe01aeef772351b2329194bb5" Oct 29 23:01:38.949734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:38.949686 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerName="payload-pre-processing" containerID="cri-o://91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb" gracePeriod=25 Oct 29 23:01:39.224545 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:39.224464 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" path="/var/lib/kubelet/pods/933caef7-daeb-4332-97a6-5ea2aedfe708/volumes" Oct 29 23:01:39.514800 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:39.514689 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:39.515281 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:39.514799 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:43.162728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:43.162672 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj"] Oct 29 23:01:47.782185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:47.782145 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49"] Oct 29 23:01:47.793206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:47.793173 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-afdbe8-7xp49"] Oct 29 23:01:48.163229 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:48.163180 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerName="payload-pre-processing" containerID="cri-o://db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d" gracePeriod=25 Oct 29 23:01:49.237801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:49.237765 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affb3e9a-34f6-4159-b704-b9d0442397c9" path="/var/lib/kubelet/pods/affb3e9a-34f6-4159-b704-b9d0442397c9/volumes" Oct 29 23:01:49.515002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:49.514915 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:49.515002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:49.514929 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:52.368710 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:52.368656 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:01:54.197897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:54.197862 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:56.879492 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:56.879455 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll"] Oct 29 23:01:56.879979 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:56.879732 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" containerID="cri-o://957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a" gracePeriod=30 Oct 29 23:01:57.452851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.452811 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx"] Oct 29 23:01:57.453665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.453640 5367 cpu_manager.go:414] "containerMap: removing container" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" Oct 29 23:01:57.453665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.453666 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" Oct 29 23:01:57.453893 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.453686 5367 cpu_manager.go:414] "containerMap: removing container" podUID="affb3e9a-34f6-4159-b704-b9d0442397c9" containerName="revoke-keys" Oct 29 23:01:57.453893 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.453693 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affb3e9a-34f6-4159-b704-b9d0442397c9" containerName="revoke-keys" Oct 29 23:01:57.454014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.453994 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" Oct 29 23:01:57.454066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.454016 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="affb3e9a-34f6-4159-b704-b9d0442397c9" containerName="revoke-keys" Oct 29 23:01:57.460450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.460428 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.481855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.481821 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:01:57.487189 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.487157 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx"] Oct 29 23:01:57.492355 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.492325 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:01:57.525675 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.525625 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dls7c\" (UniqueName: \"kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.525895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.525831 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.526008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.525983 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.627143 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.627105 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dls7c\" (UniqueName: \"kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.627337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.627216 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.627337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.627305 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.627927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.627907 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.629843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.629826 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.648108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.648079 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dls7c\" (UniqueName: \"kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:57.777153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:57.777046 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:01:58.112194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:58.112151 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx"] Oct 29 23:01:58.112532 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:01:58.112326 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c622489_7716_4a6c_a451_5bb300448e4d.slice/crio-9501f3418bc8a3d717c947d5a04235113b57ff1a300e2910c1f24f3bb3339c92 WatchSource:0}: Error finding container 9501f3418bc8a3d717c947d5a04235113b57ff1a300e2910c1f24f3bb3339c92: Status 404 returned error can't find the container with id 9501f3418bc8a3d717c947d5a04235113b57ff1a300e2910c1f24f3bb3339c92 Oct 29 23:01:58.684835 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:58.684733 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" event={"ID":"3c622489-7716-4a6c-a451-5bb300448e4d","Type":"ContainerStarted","Data":"984c9e2b76f4c2cc39d9955b427be367faedb940edf99f30b160f24b6649bafd"} Oct 29 23:01:58.685002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:58.684838 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" event={"ID":"3c622489-7716-4a6c-a451-5bb300448e4d","Type":"ContainerStarted","Data":"9501f3418bc8a3d717c947d5a04235113b57ff1a300e2910c1f24f3bb3339c92"} Oct 29 23:01:58.749843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:58.749780 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" podStartSLOduration=1.749758791 podStartE2EDuration="1.749758791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:58.731237017 +0000 UTC m=+2785.625311942" watchObservedRunningTime="2026-10-29 23:01:58.749758791 +0000 UTC m=+2785.643833712" Oct 29 23:01:59.229741 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.229679 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 23:01:59.261366 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.261340 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 23:01:59.354512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.354468 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687\" (UniqueName: \"kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687\") pod \"9e4da448-aa87-450e-b67b-b5bf84787c26\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " Oct 29 23:01:59.354711 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.354556 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b\" (UniqueName: \"kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b\") pod \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " Oct 29 23:01:59.354711 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.354596 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume\") pod \"9e4da448-aa87-450e-b67b-b5bf84787c26\" (UID: \"9e4da448-aa87-450e-b67b-b5bf84787c26\") " Oct 29 23:01:59.354871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.354765 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume\") pod \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\" (UID: \"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd\") " Oct 29 23:01:59.355099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.355074 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume" pod "9e4da448-aa87-450e-b67b-b5bf84787c26" (UID: "9e4da448-aa87-450e-b67b-b5bf84787c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:59.355199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.355163 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume" pod "fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" (UID: "fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:59.356434 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.356408 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4da448-aa87-450e-b67b-b5bf84787c26-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:59.356570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.356555 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:59.357351 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.357322 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687" pod "9e4da448-aa87-450e-b67b-b5bf84787c26" (UID: "9e4da448-aa87-450e-b67b-b5bf84787c26"). InnerVolumeSpecName "kube-api-access-w2687". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:59.357467 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.357363 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b" pod "fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" (UID: "fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd"). InnerVolumeSpecName "kube-api-access-2bb2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:59.458563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.458479 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2687\" (UniqueName: \"kubernetes.io/projected/9e4da448-aa87-450e-b67b-b5bf84787c26-kube-api-access-w2687\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:59.458563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.458511 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bb2b\" (UniqueName: \"kubernetes.io/projected/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd-kube-api-access-2bb2b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:01:59.694803 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.694749 5367 generic.go:356] "Generic (PLEG): container finished" podID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerID="642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407" exitCode=0 Oct 29 23:01:59.694803 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.694797 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" Oct 29 23:01:59.695055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.694836 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" event={"ID":"9e4da448-aa87-450e-b67b-b5bf84787c26","Type":"ContainerDied","Data":"642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407"} Oct 29 23:01:59.695055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.694889 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587" event={"ID":"9e4da448-aa87-450e-b67b-b5bf84787c26","Type":"ContainerDied","Data":"284913f6515bfe27f21e3c77496634eda67658845ea5a236d0007d1632ca8f46"} Oct 29 23:01:59.695055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.694908 5367 scope.go:122] "RemoveContainer" containerID="642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407" Oct 29 23:01:59.696692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.696670 5367 generic.go:356] "Generic (PLEG): container finished" podID="3c622489-7716-4a6c-a451-5bb300448e4d" containerID="984c9e2b76f4c2cc39d9955b427be367faedb940edf99f30b160f24b6649bafd" exitCode=0 Oct 29 23:01:59.696692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.696743 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" event={"ID":"3c622489-7716-4a6c-a451-5bb300448e4d","Type":"ContainerDied","Data":"984c9e2b76f4c2cc39d9955b427be367faedb940edf99f30b160f24b6649bafd"} Oct 29 23:01:59.698540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.698517 5367 generic.go:356] "Generic (PLEG): container finished" podID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerID="91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb" exitCode=0 Oct 29 23:01:59.698674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.698542 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" event={"ID":"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd","Type":"ContainerDied","Data":"91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb"} Oct 29 23:01:59.698674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.698573 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" Oct 29 23:01:59.698674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.698582 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn" event={"ID":"fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd","Type":"ContainerDied","Data":"8014e501c98b81548e6e44d35df641bca9933b9988183f6aa7186ffd306bd868"} Oct 29 23:01:59.708483 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.708410 5367 scope.go:122] "RemoveContainer" containerID="642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407" Oct 29 23:01:59.708816 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:59.708757 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407\": container with ID starting with 642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407 not found: ID does not exist" containerID="642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407" Oct 29 23:01:59.708816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.708802 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407"} err="failed to get container status \"642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407\": rpc error: code = NotFound desc = could not find container \"642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407\": container with ID starting with 642c1e234d7f3b2d1ef4d2165fe38425e6a45a4bcf277e8def63c3a3dc9ae407 not found: ID does not exist" Oct 29 23:01:59.708972 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.708825 5367 scope.go:122] "RemoveContainer" containerID="91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb" Oct 29 23:01:59.719605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.719582 5367 scope.go:122] "RemoveContainer" containerID="91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb" Oct 29 23:01:59.719888 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:01:59.719868 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb\": container with ID starting with 91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb not found: ID does not exist" containerID="91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb" Oct 29 23:01:59.719956 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.719896 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb"} err="failed to get container status \"91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb\": rpc error: code = NotFound desc = could not find container \"91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb\": container with ID starting with 91e98a27c64a28721ca851395064346d0d8bfa395da15b1c175c7e46f59db7cb not found: ID does not exist" Oct 29 23:01:59.751520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.751484 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn"] Oct 29 23:01:59.757268 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.757238 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e803b6-75b549ddf5-wgrwn"] Oct 29 23:01:59.793657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.793598 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587"] Oct 29 23:01:59.798078 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:01:59.798054 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e803b6-5cd68547cb-5j587"] Oct 29 23:02:00.872939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.872916 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:02:00.990182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.990087 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c\" (UniqueName: \"kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c\") pod \"3c622489-7716-4a6c-a451-5bb300448e4d\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " Oct 29 23:02:00.990182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.990134 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca\") pod \"3c622489-7716-4a6c-a451-5bb300448e4d\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " Oct 29 23:02:00.990457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.990324 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca\") pod \"3c622489-7716-4a6c-a451-5bb300448e4d\" (UID: \"3c622489-7716-4a6c-a451-5bb300448e4d\") " Oct 29 23:02:00.990576 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.990550 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca" pod "3c622489-7716-4a6c-a451-5bb300448e4d" (UID: "3c622489-7716-4a6c-a451-5bb300448e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:00.991060 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.991032 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:00.992823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.992795 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca" pod "3c622489-7716-4a6c-a451-5bb300448e4d" (UID: "3c622489-7716-4a6c-a451-5bb300448e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:00.992924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:00.992822 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c" pod "3c622489-7716-4a6c-a451-5bb300448e4d" (UID: "3c622489-7716-4a6c-a451-5bb300448e4d"). InnerVolumeSpecName "kube-api-access-dls7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:01.092437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.092401 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dls7c\" (UniqueName: \"kubernetes.io/projected/3c622489-7716-4a6c-a451-5bb300448e4d-kube-api-access-dls7c\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:01.092437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.092434 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622489-7716-4a6c-a451-5bb300448e4d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:01.223476 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.223442 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" path="/var/lib/kubelet/pods/9e4da448-aa87-450e-b67b-b5bf84787c26/volumes" Oct 29 23:02:01.223999 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.223984 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" path="/var/lib/kubelet/pods/fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd/volumes" Oct 29 23:02:01.720218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.720190 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" Oct 29 23:02:01.720506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.720223 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx" event={"ID":"3c622489-7716-4a6c-a451-5bb300448e4d","Type":"ContainerDied","Data":"9501f3418bc8a3d717c947d5a04235113b57ff1a300e2910c1f24f3bb3339c92"} Oct 29 23:02:01.720506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:01.720277 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9501f3418bc8a3d717c947d5a04235113b57ff1a300e2910c1f24f3bb3339c92" Oct 29 23:02:02.218578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.218550 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 23:02:02.315027 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.314992 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315099 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315120 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315164 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315220 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315327 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315373 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315364 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.315560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315406 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.315560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315511 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert\") pod \"4887aad4-4623-463e-a66c-f470439dc173\" (UID: \"4887aad4-4623-463e-a66c-f470439dc173\") " Oct 29 23:02:02.315894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315651 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.315894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315672 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.315894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.315854 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:02.316294 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.316272 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.316294 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.316294 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.316443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.316303 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.316443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.316312 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.316443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.316320 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4887aad4-4623-463e-a66c-f470439dc173-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.317683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.317655 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:02.317683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.317664 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.317924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.317909 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:02.317986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.317932 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m" pod "4887aad4-4623-463e-a66c-f470439dc173" (UID: "4887aad4-4623-463e-a66c-f470439dc173"). InnerVolumeSpecName "kube-api-access-9p56m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:02.417814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.417759 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p56m\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-kube-api-access-9p56m\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.417814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.417805 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4887aad4-4623-463e-a66c-f470439dc173-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.417814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.417819 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4887aad4-4623-463e-a66c-f470439dc173-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.418075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.417833 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4887aad4-4623-463e-a66c-f470439dc173-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.730227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.730133 5367 generic.go:356] "Generic (PLEG): container finished" podID="4887aad4-4623-463e-a66c-f470439dc173" containerID="957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a" exitCode=0 Oct 29 23:02:02.730227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.730210 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" Oct 29 23:02:02.730427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.730219 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" event={"ID":"4887aad4-4623-463e-a66c-f470439dc173","Type":"ContainerDied","Data":"957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a"} Oct 29 23:02:02.730427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.730307 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll" event={"ID":"4887aad4-4623-463e-a66c-f470439dc173","Type":"ContainerDied","Data":"03b56ad1be2fa0531976f73481768ab138ab08142b9ffe8093d8a37ab382efd0"} Oct 29 23:02:02.730427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.730330 5367 scope.go:122] "RemoveContainer" containerID="957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a" Oct 29 23:02:02.743246 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.743222 5367 scope.go:122] "RemoveContainer" containerID="9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4" Oct 29 23:02:02.754294 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.754271 5367 scope.go:122] "RemoveContainer" containerID="957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a" Oct 29 23:02:02.754644 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:02.754621 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a\": container with ID starting with 957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a not found: ID does not exist" containerID="957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a" Oct 29 23:02:02.754735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.754653 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a"} err="failed to get container status \"957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a\": rpc error: code = NotFound desc = could not find container \"957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a\": container with ID starting with 957041528b62048a5e36d6c731c4f111f7e9094498ddaab99b8e1b37c6d4763a not found: ID does not exist" Oct 29 23:02:02.754735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.754673 5367 scope.go:122] "RemoveContainer" containerID="9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4" Oct 29 23:02:02.754958 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:02.754941 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4\": container with ID starting with 9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4 not found: ID does not exist" containerID="9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4" Oct 29 23:02:02.755022 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.754963 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4"} err="failed to get container status \"9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4\": rpc error: code = NotFound desc = could not find container \"9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4\": container with ID starting with 9e3073f280f1e9b1708f301c4bb7cdd23a1959541a2f13ba125f13b6514737a4 not found: ID does not exist" Oct 29 23:02:02.783799 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.783766 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll"] Oct 29 23:02:02.789223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:02.789199 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-bjtll"] Oct 29 23:02:03.224407 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:03.224375 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4887aad4-4623-463e-a66c-f470439dc173" path="/var/lib/kubelet/pods/4887aad4-4623-463e-a66c-f470439dc173/volumes" Oct 29 23:02:04.198050 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:04.198012 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:07.359195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:07.359151 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn"] Oct 29 23:02:07.367814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:07.367787 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e803b6-ltzmn"] Oct 29 23:02:07.892387 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:07.892354 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb"] Oct 29 23:02:08.460187 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.460155 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 23:02:08.581935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.581898 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume\") pod \"d3d2ad78-ba6d-4c83-881e-54296aff2673\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " Oct 29 23:02:08.582108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.582071 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr\" (UniqueName: \"kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr\") pod \"d3d2ad78-ba6d-4c83-881e-54296aff2673\" (UID: \"d3d2ad78-ba6d-4c83-881e-54296aff2673\") " Oct 29 23:02:08.582314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.582296 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume" pod "d3d2ad78-ba6d-4c83-881e-54296aff2673" (UID: "d3d2ad78-ba6d-4c83-881e-54296aff2673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:08.582790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.582768 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d2ad78-ba6d-4c83-881e-54296aff2673-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:08.584524 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.584502 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr" pod "d3d2ad78-ba6d-4c83-881e-54296aff2673" (UID: "d3d2ad78-ba6d-4c83-881e-54296aff2673"). InnerVolumeSpecName "kube-api-access-28tjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:08.686816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.686693 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tjr\" (UniqueName: \"kubernetes.io/projected/d3d2ad78-ba6d-4c83-881e-54296aff2673-kube-api-access-28tjr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:08.828205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.828155 5367 generic.go:356] "Generic (PLEG): container finished" podID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerID="db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d" exitCode=0 Oct 29 23:02:08.828415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.828285 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" event={"ID":"d3d2ad78-ba6d-4c83-881e-54296aff2673","Type":"ContainerDied","Data":"db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d"} Oct 29 23:02:08.828415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.828356 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" event={"ID":"d3d2ad78-ba6d-4c83-881e-54296aff2673","Type":"ContainerDied","Data":"6ab0e372c669a526e8c487dc7b131b4b0bd2828ce5b04898e3476b34fe727fc4"} Oct 29 23:02:08.828415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.828386 5367 scope.go:122] "RemoveContainer" containerID="db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d" Oct 29 23:02:08.828624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.828472 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj" Oct 29 23:02:08.845621 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.845600 5367 scope.go:122] "RemoveContainer" containerID="db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d" Oct 29 23:02:08.845987 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:08.845969 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d\": container with ID starting with db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d not found: ID does not exist" containerID="db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d" Oct 29 23:02:08.846076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.846001 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d"} err="failed to get container status \"db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d\": rpc error: code = NotFound desc = could not find container \"db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d\": container with ID starting with db6b8512cadbd59e9990c5c301fe5fcc2159654dd8326ce96f197a3b0c7d150d not found: ID does not exist" Oct 29 23:02:08.889176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.889140 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj"] Oct 29 23:02:08.901107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:08.901078 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-afdbe8-6665895cf5-hmzxj"] Oct 29 23:02:09.225762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:09.225726 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cf1159-97a4-4021-87b8-554cfe5e1519" path="/var/lib/kubelet/pods/75cf1159-97a4-4021-87b8-554cfe5e1519/volumes" Oct 29 23:02:09.226526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:09.226504 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" path="/var/lib/kubelet/pods/d3d2ad78-ba6d-4c83-881e-54296aff2673/volumes" Oct 29 23:02:12.893053 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:12.892991 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerName="payload-pre-processing" containerID="cri-o://3cb638d99d8b13bba67138dc62a3bc69285b5503b5a84e46836a0bbf5dc2f17c" gracePeriod=25 Oct 29 23:02:14.478654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:14.478460 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk"] Oct 29 23:02:14.479430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:14.479396 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" containerID="cri-o://8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501" gracePeriod=30 Oct 29 23:02:15.103733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.103669 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn"] Oct 29 23:02:15.104380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104365 5367 cpu_manager.go:414] "containerMap: removing container" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" Oct 29 23:02:15.104423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104382 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" Oct 29 23:02:15.104423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104404 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerName="payload-processing" Oct 29 23:02:15.104423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104409 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerName="payload-processing" Oct 29 23:02:15.104423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104422 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104427 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104447 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104453 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104466 5367 cpu_manager.go:414] "containerMap: removing container" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerName="payload-pre-processing" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104471 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerName="payload-pre-processing" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104483 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3c622489-7716-4a6c-a451-5bb300448e4d" containerName="revoke-keys" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104487 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c622489-7716-4a6c-a451-5bb300448e4d" containerName="revoke-keys" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104493 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerName="payload-pre-processing" Oct 29 23:02:15.104554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104498 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerName="payload-pre-processing" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104598 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4da448-aa87-450e-b67b-b5bf84787c26" containerName="payload-processing" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104606 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104615 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4887aad4-4623-463e-a66c-f470439dc173" containerName="istio-proxy" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104622 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6ba3b6-78f4-409c-8bfa-aea95ddb30dd" containerName="payload-pre-processing" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104629 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="933caef7-daeb-4332-97a6-5ea2aedfe708" containerName="istio-proxy" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104635 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c622489-7716-4a6c-a451-5bb300448e4d" containerName="revoke-keys" Oct 29 23:02:15.104850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.104641 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d2ad78-ba6d-4c83-881e-54296aff2673" containerName="payload-pre-processing" Oct 29 23:02:15.109650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.109632 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.130954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.130932 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn"] Oct 29 23:02:15.148228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.148202 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:15.148395 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.148319 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:02:15.176308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.176280 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqrc\" (UniqueName: \"kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.176429 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.176320 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.176525 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.176499 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.278484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.278447 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqrc\" (UniqueName: \"kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.278653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.278501 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.278653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.278604 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.279296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.279275 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.281011 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.280987 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.298368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.298334 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqrc\" (UniqueName: \"kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc\") pod \"maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.424568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.424463 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:15.727812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.727783 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn"] Oct 29 23:02:15.728167 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:02:15.728140 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf566df7_1632_4f99_b188_ded2c6d79545.slice/crio-689dd450f0240538ec86afd3cd4e3b6319a1b7dfbcdaf06342a9277a1cedc8f4 WatchSource:0}: Error finding container 689dd450f0240538ec86afd3cd4e3b6319a1b7dfbcdaf06342a9277a1cedc8f4: Status 404 returned error can't find the container with id 689dd450f0240538ec86afd3cd4e3b6319a1b7dfbcdaf06342a9277a1cedc8f4 Oct 29 23:02:15.997294 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:15.997216 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" event={"ID":"bf566df7-1632-4f99-b188-ded2c6d79545","Type":"ContainerStarted","Data":"689dd450f0240538ec86afd3cd4e3b6319a1b7dfbcdaf06342a9277a1cedc8f4"} Oct 29 23:02:17.007170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:17.007096 5367 generic.go:356] "Generic (PLEG): container finished" podID="bf566df7-1632-4f99-b188-ded2c6d79545" containerID="9c47710f22b91a462ab52c1d6e1cb78b5ec5a0449a6a35b968b74f7e70753a7f" exitCode=0 Oct 29 23:02:17.007528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:17.007183 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" event={"ID":"bf566df7-1632-4f99-b188-ded2c6d79545","Type":"ContainerDied","Data":"9c47710f22b91a462ab52c1d6e1cb78b5ec5a0449a6a35b968b74f7e70753a7f"} Oct 29 23:02:18.207960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.207929 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:18.321790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.321754 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca\") pod \"bf566df7-1632-4f99-b188-ded2c6d79545\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " Oct 29 23:02:18.322041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.321984 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca\") pod \"bf566df7-1632-4f99-b188-ded2c6d79545\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " Oct 29 23:02:18.322418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.322383 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca" pod "bf566df7-1632-4f99-b188-ded2c6d79545" (UID: "bf566df7-1632-4f99-b188-ded2c6d79545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:18.322635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.322612 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc\" (UniqueName: \"kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc\") pod \"bf566df7-1632-4f99-b188-ded2c6d79545\" (UID: \"bf566df7-1632-4f99-b188-ded2c6d79545\") " Oct 29 23:02:18.323447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.323422 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:18.324645 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.324620 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca" pod "bf566df7-1632-4f99-b188-ded2c6d79545" (UID: "bf566df7-1632-4f99-b188-ded2c6d79545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:18.326253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.326232 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc" pod "bf566df7-1632-4f99-b188-ded2c6d79545" (UID: "bf566df7-1632-4f99-b188-ded2c6d79545"). InnerVolumeSpecName "kube-api-access-9cqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:18.424674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.424642 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cqrc\" (UniqueName: \"kubernetes.io/projected/bf566df7-1632-4f99-b188-ded2c6d79545-kube-api-access-9cqrc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:18.424674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:18.424669 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf566df7-1632-4f99-b188-ded2c6d79545-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.024963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.024930 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" event={"ID":"bf566df7-1632-4f99-b188-ded2c6d79545","Type":"ContainerDied","Data":"689dd450f0240538ec86afd3cd4e3b6319a1b7dfbcdaf06342a9277a1cedc8f4"} Oct 29 23:02:19.024963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.024950 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn" Oct 29 23:02:19.025176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.024982 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="689dd450f0240538ec86afd3cd4e3b6319a1b7dfbcdaf06342a9277a1cedc8f4" Oct 29 23:02:19.780105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.780075 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845011 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845200 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845251 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845285 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845328 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845369 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845447 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845544 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845622 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs\") pod \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\" (UID: \"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd\") " Oct 29 23:02:19.846313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.845620 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:19.847523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.846265 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.847771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.847742 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.847957 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.847938 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.848674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.848491 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:19.849190 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.848878 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.849293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.849233 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.861142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.861114 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:19.861265 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.861204 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.861346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.861301 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz" pod "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" (UID: "e86fe3ea-1ba9-4b52-9020-6207ebb7bebd"). InnerVolumeSpecName "kube-api-access-n5qjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:19.909642 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.909609 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p"] Oct 29 23:02:19.922193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.922156 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp"] Oct 29 23:02:19.942621 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.942593 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj"] Oct 29 23:02:19.948010 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.947984 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948010 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948011 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948026 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qjz\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-kube-api-access-n5qjz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948041 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948054 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948065 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948077 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.948191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:19.948089 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.035066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.034982 5367 generic.go:356] "Generic (PLEG): container finished" podID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerID="8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501" exitCode=0 Oct 29 23:02:20.035066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.035034 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" Oct 29 23:02:20.035308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.035068 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" event={"ID":"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd","Type":"ContainerDied","Data":"8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501"} Oct 29 23:02:20.035308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.035124 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk" event={"ID":"e86fe3ea-1ba9-4b52-9020-6207ebb7bebd","Type":"ContainerDied","Data":"32f994acd72e4548d4b5ba3f502a91d28248f52cf20026e34bdff34c4d714743"} Oct 29 23:02:20.035308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.035149 5367 scope.go:122] "RemoveContainer" containerID="8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501" Oct 29 23:02:20.055228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.051504 5367 scope.go:122] "RemoveContainer" containerID="8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501" Oct 29 23:02:20.057358 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:20.055801 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501\": container with ID starting with 8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501 not found: ID does not exist" containerID="8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501" Oct 29 23:02:20.057358 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.055955 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501"} err="failed to get container status \"8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501\": rpc error: code = NotFound desc = could not find container \"8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501\": container with ID starting with 8ff02edf8667152eaeb4da0cfacb5ba0235932e28c7f04dc3803e2b9abd2d501 not found: ID does not exist" Oct 29 23:02:20.099798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.099773 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk"] Oct 29 23:02:20.106600 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:20.106577 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-vcbkk"] Oct 29 23:02:21.121271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:21.121236 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:21.228228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:21.228198 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" path="/var/lib/kubelet/pods/e86fe3ea-1ba9-4b52-9020-6207ebb7bebd/volumes" Oct 29 23:02:22.331910 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:22.331879 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:02:23.183407 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:23.183377 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w"] Oct 29 23:02:23.228927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:23.228897 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc"] Oct 29 23:02:25.579075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:25.579040 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf"] Oct 29 23:02:25.614905 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:25.614876 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj"] Oct 29 23:02:28.184564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.184524 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" podUID="8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" containerName="payload-processing" containerID="cri-o://287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e" gracePeriod=25 Oct 29 23:02:28.229568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.229538 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" podUID="dff9a057-03e0-4b3a-837f-3fc98618e7eb" containerName="payload-pre-processing" containerID="cri-o://e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e" gracePeriod=25 Oct 29 23:02:28.512053 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.512025 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:02:28.529628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.529602 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7\" (UniqueName: \"kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7\") pod \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " Oct 29 23:02:28.529768 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.529659 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume\") pod \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\" (UID: \"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf\") " Oct 29 23:02:28.530063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.530040 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume" pod "8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" (UID: "8f27ecfa-32cf-4baa-a051-038e9ea1ceaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:28.530171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.530155 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:28.531880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.531854 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7" pod "8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" (UID: "8f27ecfa-32cf-4baa-a051-038e9ea1ceaf"). InnerVolumeSpecName "kube-api-access-s7wm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:28.548627 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.548604 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:02:28.631146 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.631111 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99\" (UniqueName: \"kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99\") pod \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " Oct 29 23:02:28.631341 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.631267 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume\") pod \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\" (UID: \"dff9a057-03e0-4b3a-837f-3fc98618e7eb\") " Oct 29 23:02:28.631747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.631688 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume" pod "dff9a057-03e0-4b3a-837f-3fc98618e7eb" (UID: "dff9a057-03e0-4b3a-837f-3fc98618e7eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:28.632071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.632046 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wm7\" (UniqueName: \"kubernetes.io/projected/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf-kube-api-access-s7wm7\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:28.632172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.632077 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff9a057-03e0-4b3a-837f-3fc98618e7eb-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:28.633485 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.633461 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99" pod "dff9a057-03e0-4b3a-837f-3fc98618e7eb" (UID: "dff9a057-03e0-4b3a-837f-3fc98618e7eb"). InnerVolumeSpecName "kube-api-access-g8l99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:28.733208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:28.733122 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8l99\" (UniqueName: \"kubernetes.io/projected/dff9a057-03e0-4b3a-837f-3fc98618e7eb-kube-api-access-g8l99\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.115844 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.115808 5367 generic.go:356] "Generic (PLEG): container finished" podID="8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" containerID="287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e" exitCode=0 Oct 29 23:02:29.116034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.115856 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" Oct 29 23:02:29.116034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.115883 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" event={"ID":"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf","Type":"ContainerDied","Data":"287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e"} Oct 29 23:02:29.116034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.115938 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w" event={"ID":"8f27ecfa-32cf-4baa-a051-038e9ea1ceaf","Type":"ContainerDied","Data":"6b0705e4759e4836040fd25c7643a69ebbc79cc26eb03821d3ecf8a47d5035d6"} Oct 29 23:02:29.116034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.115960 5367 scope.go:122] "RemoveContainer" containerID="287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e" Oct 29 23:02:29.117626 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.117600 5367 generic.go:356] "Generic (PLEG): container finished" podID="dff9a057-03e0-4b3a-837f-3fc98618e7eb" containerID="e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e" exitCode=0 Oct 29 23:02:29.117824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.117641 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" Oct 29 23:02:29.117824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.117657 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" event={"ID":"dff9a057-03e0-4b3a-837f-3fc98618e7eb","Type":"ContainerDied","Data":"e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e"} Oct 29 23:02:29.117824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.117780 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc" event={"ID":"dff9a057-03e0-4b3a-837f-3fc98618e7eb","Type":"ContainerDied","Data":"b4e6b48287b996045b2b9e50545460a8f4e410af14faebfc6dc1c157b8b4eabb"} Oct 29 23:02:29.128592 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.128569 5367 scope.go:122] "RemoveContainer" containerID="287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e" Oct 29 23:02:29.129008 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:29.128987 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e\": container with ID starting with 287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e not found: ID does not exist" containerID="287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e" Oct 29 23:02:29.129126 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.129019 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e"} err="failed to get container status \"287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e\": rpc error: code = NotFound desc = could not find container \"287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e\": container with ID starting with 287e849bad1b18335345eea1c9b325446da6780dc87733e5fb404c0a7245481e not found: ID does not exist" Oct 29 23:02:29.129126 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.129043 5367 scope.go:122] "RemoveContainer" containerID="e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e" Oct 29 23:02:29.142895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.142877 5367 scope.go:122] "RemoveContainer" containerID="e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e" Oct 29 23:02:29.143186 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:29.143167 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e\": container with ID starting with e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e not found: ID does not exist" containerID="e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e" Oct 29 23:02:29.143293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.143191 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e"} err="failed to get container status \"e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e\": rpc error: code = NotFound desc = could not find container \"e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e\": container with ID starting with e275b25a4aeeb78a2e3582fe621367d18444e93d7b2b7a182c7a314a1d8f5e1e not found: ID does not exist" Oct 29 23:02:29.173735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.173014 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w"] Oct 29 23:02:29.184540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.184521 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-5746854446-rqk4w"] Oct 29 23:02:29.213318 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.213298 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc"] Oct 29 23:02:29.229598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.229567 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" path="/var/lib/kubelet/pods/8f27ecfa-32cf-4baa-a051-038e9ea1ceaf/volumes" Oct 29 23:02:29.230076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:29.230049 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-67cb496b87-mjvbc"] Oct 29 23:02:30.580058 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:30.579995 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" containerName="payload-processing" containerID="cri-o://8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1" gracePeriod=25 Oct 29 23:02:30.616154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:30.616104 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerName="payload-pre-processing" containerID="cri-o://a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f" gracePeriod=25 Oct 29 23:02:31.120895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:31.120856 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:31.223041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:31.223007 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff9a057-03e0-4b3a-837f-3fc98618e7eb" path="/var/lib/kubelet/pods/dff9a057-03e0-4b3a-837f-3fc98618e7eb/volumes" Oct 29 23:02:32.578750 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:32.578690 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:32.589898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:32.589856 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:33.155653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.155568 5367 generic.go:356] "Generic (PLEG): container finished" podID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerID="3cb638d99d8b13bba67138dc62a3bc69285b5503b5a84e46836a0bbf5dc2f17c" exitCode=0 Oct 29 23:02:33.155882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.155643 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" event={"ID":"047b35a0-9d90-4bc0-a00c-95cdc2c191e2","Type":"ContainerDied","Data":"3cb638d99d8b13bba67138dc62a3bc69285b5503b5a84e46836a0bbf5dc2f17c"} Oct 29 23:02:33.184183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.184158 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 23:02:33.204030 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.204005 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume\") pod \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " Oct 29 23:02:33.204136 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.204108 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6\" (UniqueName: \"kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6\") pod \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\" (UID: \"047b35a0-9d90-4bc0-a00c-95cdc2c191e2\") " Oct 29 23:02:33.204347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.204325 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume" pod "047b35a0-9d90-4bc0-a00c-95cdc2c191e2" (UID: "047b35a0-9d90-4bc0-a00c-95cdc2c191e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:33.204793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.204777 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:33.206085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.206065 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6" pod "047b35a0-9d90-4bc0-a00c-95cdc2c191e2" (UID: "047b35a0-9d90-4bc0-a00c-95cdc2c191e2"). InnerVolumeSpecName "kube-api-access-v8dx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:33.306963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.306925 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dx6\" (UniqueName: \"kubernetes.io/projected/047b35a0-9d90-4bc0-a00c-95cdc2c191e2-kube-api-access-v8dx6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:33.926331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.926297 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h"] Oct 29 23:02:33.926969 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.926954 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bf566df7-1632-4f99-b188-ded2c6d79545" containerName="revoke-keys" Oct 29 23:02:33.927034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.926971 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf566df7-1632-4f99-b188-ded2c6d79545" containerName="revoke-keys" Oct 29 23:02:33.927034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.926987 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" containerName="payload-processing" Oct 29 23:02:33.927034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.926991 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" containerName="payload-processing" Oct 29 23:02:33.927034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927002 5367 cpu_manager.go:414] "containerMap: removing container" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerName="payload-pre-processing" Oct 29 23:02:33.927034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927008 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerName="payload-pre-processing" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927048 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927053 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927061 5367 cpu_manager.go:414] "containerMap: removing container" podUID="dff9a057-03e0-4b3a-837f-3fc98618e7eb" containerName="payload-pre-processing" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927066 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff9a057-03e0-4b3a-837f-3fc98618e7eb" containerName="payload-pre-processing" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927164 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf566df7-1632-4f99-b188-ded2c6d79545" containerName="revoke-keys" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927171 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f27ecfa-32cf-4baa-a051-038e9ea1ceaf" containerName="payload-processing" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927178 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" containerName="payload-pre-processing" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927193 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86fe3ea-1ba9-4b52-9020-6207ebb7bebd" containerName="istio-proxy" Oct 29 23:02:33.927199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.927198 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff9a057-03e0-4b3a-837f-3fc98618e7eb" containerName="payload-pre-processing" Oct 29 23:02:33.930859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.930837 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:33.947749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.947717 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78"] Oct 29 23:02:33.951887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.951858 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:02:33.952049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.951932 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:33.953315 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:33.953292 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.023090 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.023146 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.023176 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4g4x\" (UniqueName: \"kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.023311 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lsd\" (UniqueName: \"kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.023468 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.023633 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.031734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.026809 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h"] Oct 29 23:02:34.148074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.148035 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.149503 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.149477 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.149685 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.149672 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.149859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.149842 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4g4x\" (UniqueName: \"kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.150080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.150066 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lsd\" (UniqueName: \"kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.150316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.150303 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.151328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.151305 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.152921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.149329 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.152921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.152818 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78"] Oct 29 23:02:34.154650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.154606 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.160640 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.160618 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.171724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.171224 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" event={"ID":"047b35a0-9d90-4bc0-a00c-95cdc2c191e2","Type":"ContainerDied","Data":"8ee05ddda84f9373382dd520e2acb8e097caa833136ed6284e3b24c7a3b07f4b"} Oct 29 23:02:34.171724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.171309 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb" Oct 29 23:02:34.171724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.171316 5367 scope.go:122] "RemoveContainer" containerID="3cb638d99d8b13bba67138dc62a3bc69285b5503b5a84e46836a0bbf5dc2f17c" Oct 29 23:02:34.177144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.174015 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4g4x\" (UniqueName: \"kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x\") pod \"maas-api-revoke-sub-a7f60c83-7854h\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.177144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.174475 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lsd\" (UniqueName: \"kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd\") pod \"maas-api-revoke-sub-842fd636-rgr78\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.245461 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.245419 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb"] Oct 29 23:02:34.252336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.252302 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-84304f-dc8b6ccff-gdkvb"] Oct 29 23:02:34.262120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.262096 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:34.457099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.457020 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:34.583350 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.583087 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h"] Oct 29 23:02:34.597584 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:02:34.597551 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294ad7f3_5730_49a7_85d0_19ff1b8b13b8.slice/crio-0ecb918f2abb805520b81fd815ac94d7c56cc1537fea78dfb600257d6ebec82b WatchSource:0}: Error finding container 0ecb918f2abb805520b81fd815ac94d7c56cc1537fea78dfb600257d6ebec82b: Status 404 returned error can't find the container with id 0ecb918f2abb805520b81fd815ac94d7c56cc1537fea78dfb600257d6ebec82b Oct 29 23:02:34.759834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.759798 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78"] Oct 29 23:02:34.856498 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:02:34.856465 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1421c097_e373_4efa_9b49_256eb3438c6a.slice/crio-886f68d20c6a79b4f630d4defa9da1dfa83b72089668e5c0e7c876a6a9fdd16e WatchSource:0}: Error finding container 886f68d20c6a79b4f630d4defa9da1dfa83b72089668e5c0e7c876a6a9fdd16e: Status 404 returned error can't find the container with id 886f68d20c6a79b4f630d4defa9da1dfa83b72089668e5c0e7c876a6a9fdd16e Oct 29 23:02:34.932349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.932316 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" containerID="cri-o://f154416c30e87d697094fc766ca80b00e84b1ff088ac105ea06b5fa92b376194" gracePeriod=45 Oct 29 23:02:34.936723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.936674 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": EOF" Oct 29 23:02:34.944276 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.944243 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" containerID="cri-o://a6ee479461cc4acc37ab912c0142b8a90bb01ff3580d0c3d2bb453c14643b413" gracePeriod=45 Oct 29 23:02:34.969463 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:34.969412 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" containerID="cri-o://8a9d94a3ec67d6063863bb8419713e2cd8cf30f9470587f8403d1b81a4d2bf63" gracePeriod=45 Oct 29 23:02:35.172476 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.172434 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused" Oct 29 23:02:35.188678 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.188653 5367 generic.go:356] "Generic (PLEG): container finished" podID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerID="f154416c30e87d697094fc766ca80b00e84b1ff088ac105ea06b5fa92b376194" exitCode=0 Oct 29 23:02:35.188827 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.188807 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" event={"ID":"b5c9e995-c1bf-4f07-ac73-73248472db5d","Type":"ContainerDied","Data":"f154416c30e87d697094fc766ca80b00e84b1ff088ac105ea06b5fa92b376194"} Oct 29 23:02:35.190832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.190809 5367 generic.go:356] "Generic (PLEG): container finished" podID="378e0f62-b817-47b8-ade5-40585173dc5d" containerID="a6ee479461cc4acc37ab912c0142b8a90bb01ff3580d0c3d2bb453c14643b413" exitCode=0 Oct 29 23:02:35.190971 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.190873 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" event={"ID":"378e0f62-b817-47b8-ade5-40585173dc5d","Type":"ContainerDied","Data":"a6ee479461cc4acc37ab912c0142b8a90bb01ff3580d0c3d2bb453c14643b413"} Oct 29 23:02:35.192504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.192477 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" event={"ID":"294ad7f3-5730-49a7-85d0-19ff1b8b13b8","Type":"ContainerStarted","Data":"0ecb918f2abb805520b81fd815ac94d7c56cc1537fea78dfb600257d6ebec82b"} Oct 29 23:02:35.196733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.193893 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" event={"ID":"1421c097-e373-4efa-9b49-256eb3438c6a","Type":"ContainerStarted","Data":"886f68d20c6a79b4f630d4defa9da1dfa83b72089668e5c0e7c876a6a9fdd16e"} Oct 29 23:02:35.196733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.195715 5367 generic.go:356] "Generic (PLEG): container finished" podID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerID="8a9d94a3ec67d6063863bb8419713e2cd8cf30f9470587f8403d1b81a4d2bf63" exitCode=0 Oct 29 23:02:35.196733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.195861 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" event={"ID":"9cb0eb6c-2930-431f-a36d-c6f3dda9b228","Type":"ContainerDied","Data":"8a9d94a3ec67d6063863bb8419713e2cd8cf30f9470587f8403d1b81a4d2bf63"} Oct 29 23:02:35.227398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.227366 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047b35a0-9d90-4bc0-a00c-95cdc2c191e2" path="/var/lib/kubelet/pods/047b35a0-9d90-4bc0-a00c-95cdc2c191e2/volumes" Oct 29 23:02:35.272878 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.269550 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 23:02:35.334978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.334945 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 23:02:35.339258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.339228 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 23:02:35.382104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382072 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir\") pod \"378e0f62-b817-47b8-ade5-40585173dc5d\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " Oct 29 23:02:35.382264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382127 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home\") pod \"378e0f62-b817-47b8-ade5-40585173dc5d\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " Oct 29 23:02:35.382264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382167 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6\" (UniqueName: \"kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6\") pod \"378e0f62-b817-47b8-ade5-40585173dc5d\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " Oct 29 23:02:35.382340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382286 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache\") pod \"378e0f62-b817-47b8-ade5-40585173dc5d\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " Oct 29 23:02:35.382387 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382367 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs\") pod \"378e0f62-b817-47b8-ade5-40585173dc5d\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " Oct 29 23:02:35.382431 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382393 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir" pod "378e0f62-b817-47b8-ade5-40585173dc5d" (UID: "378e0f62-b817-47b8-ade5-40585173dc5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.382468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382431 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm\") pod \"378e0f62-b817-47b8-ade5-40585173dc5d\" (UID: \"378e0f62-b817-47b8-ade5-40585173dc5d\") " Oct 29 23:02:35.382468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382446 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home" pod "378e0f62-b817-47b8-ade5-40585173dc5d" (UID: "378e0f62-b817-47b8-ade5-40585173dc5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.382627 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.382606 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache" pod "378e0f62-b817-47b8-ade5-40585173dc5d" (UID: "378e0f62-b817-47b8-ade5-40585173dc5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.383109 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.383087 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.383109 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.383108 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.383294 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.383116 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.385106 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.385071 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm" pod "378e0f62-b817-47b8-ade5-40585173dc5d" (UID: "378e0f62-b817-47b8-ade5-40585173dc5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.385277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.385237 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6" pod "378e0f62-b817-47b8-ade5-40585173dc5d" (UID: "378e0f62-b817-47b8-ade5-40585173dc5d"). InnerVolumeSpecName "kube-api-access-d7vm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:35.385411 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.385381 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs" pod "378e0f62-b817-47b8-ade5-40585173dc5d" (UID: "378e0f62-b817-47b8-ade5-40585173dc5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:35.484559 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484526 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm\") pod \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " Oct 29 23:02:35.484749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484572 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home\") pod \"b5c9e995-c1bf-4f07-ac73-73248472db5d\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " Oct 29 23:02:35.484749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484654 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm\") pod \"b5c9e995-c1bf-4f07-ac73-73248472db5d\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " Oct 29 23:02:35.484749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484692 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir\") pod \"b5c9e995-c1bf-4f07-ac73-73248472db5d\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " Oct 29 23:02:35.484939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484762 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs\") pod \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " Oct 29 23:02:35.484939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484815 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache\") pod \"b5c9e995-c1bf-4f07-ac73-73248472db5d\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " Oct 29 23:02:35.484939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484845 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4\" (UniqueName: \"kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4\") pod \"b5c9e995-c1bf-4f07-ac73-73248472db5d\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " Oct 29 23:02:35.484939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.484850 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home" pod "b5c9e995-c1bf-4f07-ac73-73248472db5d" (UID: "b5c9e995-c1bf-4f07-ac73-73248472db5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.485166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485018 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache\") pod \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " Oct 29 23:02:35.485166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485046 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache" pod "b5c9e995-c1bf-4f07-ac73-73248472db5d" (UID: "b5c9e995-c1bf-4f07-ac73-73248472db5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.485166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485108 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n\" (UniqueName: \"kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n\") pod \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " Oct 29 23:02:35.485166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485145 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs\") pod \"b5c9e995-c1bf-4f07-ac73-73248472db5d\" (UID: \"b5c9e995-c1bf-4f07-ac73-73248472db5d\") " Oct 29 23:02:35.485406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485225 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir" pod "b5c9e995-c1bf-4f07-ac73-73248472db5d" (UID: "b5c9e995-c1bf-4f07-ac73-73248472db5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.485406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485263 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache" pod "9cb0eb6c-2930-431f-a36d-c6f3dda9b228" (UID: "9cb0eb6c-2930-431f-a36d-c6f3dda9b228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.485406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home\") pod \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " Oct 29 23:02:35.485406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.485361 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir\") pod \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\" (UID: \"9cb0eb6c-2930-431f-a36d-c6f3dda9b228\") " Oct 29 23:02:35.486196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486166 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e0f62-b817-47b8-ade5-40585173dc5d-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486198 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e0f62-b817-47b8-ade5-40585173dc5d-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486211 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486222 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486234 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486248 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vm6\" (UniqueName: \"kubernetes.io/projected/378e0f62-b817-47b8-ade5-40585173dc5d-kube-api-access-d7vm6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486261 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.486726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486641 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home" pod "9cb0eb6c-2930-431f-a36d-c6f3dda9b228" (UID: "9cb0eb6c-2930-431f-a36d-c6f3dda9b228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.486803 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.486747 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir" pod "9cb0eb6c-2930-431f-a36d-c6f3dda9b228" (UID: "9cb0eb6c-2930-431f-a36d-c6f3dda9b228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.487282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.487257 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4" pod "b5c9e995-c1bf-4f07-ac73-73248472db5d" (UID: "b5c9e995-c1bf-4f07-ac73-73248472db5d"). InnerVolumeSpecName "kube-api-access-vcss4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:35.487399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.487299 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs" pod "9cb0eb6c-2930-431f-a36d-c6f3dda9b228" (UID: "9cb0eb6c-2930-431f-a36d-c6f3dda9b228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:35.487399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.487307 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm" pod "b5c9e995-c1bf-4f07-ac73-73248472db5d" (UID: "b5c9e995-c1bf-4f07-ac73-73248472db5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.487399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.487312 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm" pod "9cb0eb6c-2930-431f-a36d-c6f3dda9b228" (UID: "9cb0eb6c-2930-431f-a36d-c6f3dda9b228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.487532 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.487512 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n" pod "9cb0eb6c-2930-431f-a36d-c6f3dda9b228" (UID: "9cb0eb6c-2930-431f-a36d-c6f3dda9b228"). InnerVolumeSpecName "kube-api-access-4xf4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:35.487682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.487655 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs" pod "b5c9e995-c1bf-4f07-ac73-73248472db5d" (UID: "b5c9e995-c1bf-4f07-ac73-73248472db5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:35.587446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587413 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587442 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587450 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587459 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c9e995-c1bf-4f07-ac73-73248472db5d-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587467 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587475 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcss4\" (UniqueName: \"kubernetes.io/projected/b5c9e995-c1bf-4f07-ac73-73248472db5d-kube-api-access-vcss4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587484 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xf4n\" (UniqueName: \"kubernetes.io/projected/9cb0eb6c-2930-431f-a36d-c6f3dda9b228-kube-api-access-4xf4n\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.587882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.587493 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c9e995-c1bf-4f07-ac73-73248472db5d-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.911155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.911116 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7"] Oct 29 23:02:35.944721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:35.944670 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd"] Oct 29 23:02:36.206084 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.206051 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" Oct 29 23:02:36.206084 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.206062 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p" event={"ID":"b5c9e995-c1bf-4f07-ac73-73248472db5d","Type":"ContainerDied","Data":"3d82f24eea1840939c09ecee70c1b2df5d98349c0b4e2b6226b34ac598ff6064"} Oct 29 23:02:36.206336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.206120 5367 scope.go:122] "RemoveContainer" containerID="f154416c30e87d697094fc766ca80b00e84b1ff088ac105ea06b5fa92b376194" Oct 29 23:02:36.208853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.208828 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" Oct 29 23:02:36.209008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.208830 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp" event={"ID":"378e0f62-b817-47b8-ade5-40585173dc5d","Type":"ContainerDied","Data":"40b62567f81bdde0571421317085e5ee19cf0fa0d6e6794bdbab452249331051"} Oct 29 23:02:36.210628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.210601 5367 generic.go:356] "Generic (PLEG): container finished" podID="294ad7f3-5730-49a7-85d0-19ff1b8b13b8" containerID="afec1ded198aacc40580a6c2a2fd5273265a1f51681efbbacbb4a575eb8714e0" exitCode=0 Oct 29 23:02:36.210746 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.210685 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" event={"ID":"294ad7f3-5730-49a7-85d0-19ff1b8b13b8","Type":"ContainerDied","Data":"afec1ded198aacc40580a6c2a2fd5273265a1f51681efbbacbb4a575eb8714e0"} Oct 29 23:02:36.212542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.212524 5367 generic.go:356] "Generic (PLEG): container finished" podID="1421c097-e373-4efa-9b49-256eb3438c6a" containerID="79ec362f8fa6cb8234baf1f7b30ce21bf6e453a25f5399f905f04401b927c73d" exitCode=0 Oct 29 23:02:36.212672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.212612 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" event={"ID":"1421c097-e373-4efa-9b49-256eb3438c6a","Type":"ContainerDied","Data":"79ec362f8fa6cb8234baf1f7b30ce21bf6e453a25f5399f905f04401b927c73d"} Oct 29 23:02:36.214142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.214122 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" Oct 29 23:02:36.214245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.214150 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj" event={"ID":"9cb0eb6c-2930-431f-a36d-c6f3dda9b228","Type":"ContainerDied","Data":"406fcb040c7e7a5306e919a4dce101fd3205816bead1cdabcbb1296ffccd0d1f"} Oct 29 23:02:36.219552 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.219534 5367 scope.go:122] "RemoveContainer" containerID="a6ee479461cc4acc37ab912c0142b8a90bb01ff3580d0c3d2bb453c14643b413" Oct 29 23:02:36.231642 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.231620 5367 scope.go:122] "RemoveContainer" containerID="8a9d94a3ec67d6063863bb8419713e2cd8cf30f9470587f8403d1b81a4d2bf63" Oct 29 23:02:36.384762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.381763 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp"] Oct 29 23:02:36.388927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.388895 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-w1-879596-kserve-76cf6899bd-jzfjp"] Oct 29 23:02:36.422231 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.422160 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p"] Oct 29 23:02:36.427797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.427772 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-879596/distinct-2-w1-879596-kserve-b747c7575-7gq2p"] Oct 29 23:02:36.461919 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.461890 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj"] Oct 29 23:02:36.465933 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:36.465906 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-879596/simulator-w1-879596-kserve-f7989cf76-wtgtj"] Oct 29 23:02:37.028489 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.028453 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx"] Oct 29 23:02:37.034123 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.034098 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-84304f-9wzfx"] Oct 29 23:02:37.224170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.224131 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" path="/var/lib/kubelet/pods/378e0f62-b817-47b8-ade5-40585173dc5d/volumes" Oct 29 23:02:37.224672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.224653 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c622489-7716-4a6c-a451-5bb300448e4d" path="/var/lib/kubelet/pods/3c622489-7716-4a6c-a451-5bb300448e4d/volumes" Oct 29 23:02:37.225122 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.225107 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" path="/var/lib/kubelet/pods/9cb0eb6c-2930-431f-a36d-c6f3dda9b228/volumes" Oct 29 23:02:37.225837 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.225822 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" path="/var/lib/kubelet/pods/b5c9e995-c1bf-4f07-ac73-73248472db5d/volumes" Oct 29 23:02:37.442837 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.442813 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:37.446568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.446545 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:37.485356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.485321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca\") pod \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " Oct 29 23:02:37.485580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.485382 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd\" (UniqueName: \"kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd\") pod \"1421c097-e373-4efa-9b49-256eb3438c6a\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " Oct 29 23:02:37.485580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.485477 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x\" (UniqueName: \"kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x\") pod \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " Oct 29 23:02:37.485580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.485537 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca\") pod \"1421c097-e373-4efa-9b49-256eb3438c6a\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " Oct 29 23:02:37.485793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.485580 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca\") pod \"1421c097-e373-4efa-9b49-256eb3438c6a\" (UID: \"1421c097-e373-4efa-9b49-256eb3438c6a\") " Oct 29 23:02:37.485853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.485816 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca\") pod \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\" (UID: \"294ad7f3-5730-49a7-85d0-19ff1b8b13b8\") " Oct 29 23:02:37.487037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.486221 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca" pod "1421c097-e373-4efa-9b49-256eb3438c6a" (UID: "1421c097-e373-4efa-9b49-256eb3438c6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:37.487037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.486387 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.487761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.487056 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca" pod "294ad7f3-5730-49a7-85d0-19ff1b8b13b8" (UID: "294ad7f3-5730-49a7-85d0-19ff1b8b13b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:37.489664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.489619 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca" pod "294ad7f3-5730-49a7-85d0-19ff1b8b13b8" (UID: "294ad7f3-5730-49a7-85d0-19ff1b8b13b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:37.489664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.489638 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca" pod "1421c097-e373-4efa-9b49-256eb3438c6a" (UID: "1421c097-e373-4efa-9b49-256eb3438c6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:37.490171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.489862 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd" pod "1421c097-e373-4efa-9b49-256eb3438c6a" (UID: "1421c097-e373-4efa-9b49-256eb3438c6a"). InnerVolumeSpecName "kube-api-access-z5lsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.490532 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.490510 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x" pod "294ad7f3-5730-49a7-85d0-19ff1b8b13b8" (UID: "294ad7f3-5730-49a7-85d0-19ff1b8b13b8"). InnerVolumeSpecName "kube-api-access-m4g4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.588170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.588138 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lsd\" (UniqueName: \"kubernetes.io/projected/1421c097-e373-4efa-9b49-256eb3438c6a-kube-api-access-z5lsd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.588170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.588170 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4g4x\" (UniqueName: \"kubernetes.io/projected/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-kube-api-access-m4g4x\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.588399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.588184 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1421c097-e373-4efa-9b49-256eb3438c6a-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.588399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.588207 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.588399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:37.588220 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ad7f3-5730-49a7-85d0-19ff1b8b13b8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:38.239956 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:38.239922 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" Oct 29 23:02:38.240460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:38.239949 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h" event={"ID":"294ad7f3-5730-49a7-85d0-19ff1b8b13b8","Type":"ContainerDied","Data":"0ecb918f2abb805520b81fd815ac94d7c56cc1537fea78dfb600257d6ebec82b"} Oct 29 23:02:38.240460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:38.240002 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ecb918f2abb805520b81fd815ac94d7c56cc1537fea78dfb600257d6ebec82b" Oct 29 23:02:38.241512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:38.241486 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" Oct 29 23:02:38.241646 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:38.241515 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78" event={"ID":"1421c097-e373-4efa-9b49-256eb3438c6a","Type":"ContainerDied","Data":"886f68d20c6a79b4f630d4defa9da1dfa83b72089668e5c0e7c876a6a9fdd16e"} Oct 29 23:02:38.241646 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:38.241554 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886f68d20c6a79b4f630d4defa9da1dfa83b72089668e5c0e7c876a6a9fdd16e" Oct 29 23:02:39.032437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:39.032408 5367 scope.go:122] "RemoveContainer" containerID="a9685943ddbc9eff43d0986a57a9cb38370102e8f56c9370c4d042fead1729f6" Oct 29 23:02:40.911841 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:40.911783 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" podUID="b144f021-128f-4b08-83b8-46fdb7fee0bb" containerName="payload-processing" containerID="cri-o://257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978" gracePeriod=25 Oct 29 23:02:40.945849 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:40.945808 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" podUID="6a9c8e10-9a9e-497e-85fc-e0b241c25f60" containerName="payload-pre-processing" containerID="cri-o://751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104" gracePeriod=25 Oct 29 23:02:41.248159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.248134 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:02:41.251806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.251785 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:02:41.318243 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.318210 5367 generic.go:356] "Generic (PLEG): container finished" podID="b144f021-128f-4b08-83b8-46fdb7fee0bb" containerID="257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978" exitCode=0 Oct 29 23:02:41.318450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.318260 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" Oct 29 23:02:41.318450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.318302 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" event={"ID":"b144f021-128f-4b08-83b8-46fdb7fee0bb","Type":"ContainerDied","Data":"257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978"} Oct 29 23:02:41.318450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.318373 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7" event={"ID":"b144f021-128f-4b08-83b8-46fdb7fee0bb","Type":"ContainerDied","Data":"8f6dce0f9fc7dc1adb37cad1114faa2888f39ffc4bd3101e96fff121714cb22c"} Oct 29 23:02:41.318450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.318406 5367 scope.go:122] "RemoveContainer" containerID="257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978" Oct 29 23:02:41.320452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.320433 5367 generic.go:356] "Generic (PLEG): container finished" podID="6a9c8e10-9a9e-497e-85fc-e0b241c25f60" containerID="751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104" exitCode=0 Oct 29 23:02:41.320574 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.320486 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" Oct 29 23:02:41.320669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.320501 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" event={"ID":"6a9c8e10-9a9e-497e-85fc-e0b241c25f60","Type":"ContainerDied","Data":"751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104"} Oct 29 23:02:41.320755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.320733 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd" event={"ID":"6a9c8e10-9a9e-497e-85fc-e0b241c25f60","Type":"ContainerDied","Data":"0e6325ced3cb356bab8674d58d452a6a0c8ba6e9d5c9137d056261693a704842"} Oct 29 23:02:41.330805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.330785 5367 scope.go:122] "RemoveContainer" containerID="257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978" Oct 29 23:02:41.331073 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:41.331056 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978\": container with ID starting with 257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978 not found: ID does not exist" containerID="257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978" Oct 29 23:02:41.331144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.331081 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978"} err="failed to get container status \"257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978\": rpc error: code = NotFound desc = could not find container \"257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978\": container with ID starting with 257e924061901d601e4afe038b43c9989aebdd31bd4407dae84311ebb1207978 not found: ID does not exist" Oct 29 23:02:41.331144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.331100 5367 scope.go:122] "RemoveContainer" containerID="751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104" Oct 29 23:02:41.340976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.340944 5367 scope.go:122] "RemoveContainer" containerID="751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104" Oct 29 23:02:41.341271 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:41.341252 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104\": container with ID starting with 751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104 not found: ID does not exist" containerID="751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104" Oct 29 23:02:41.341327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.341278 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104"} err="failed to get container status \"751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104\": rpc error: code = NotFound desc = could not find container \"751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104\": container with ID starting with 751b38fd460cfb9288ab19cc5e777d4a2b8c4dff0b7fb3c6b83a8eb4b6bf5104 not found: ID does not exist" Oct 29 23:02:41.341663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.341632 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume\") pod \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " Oct 29 23:02:41.341860 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.341844 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume\") pod \"b144f021-128f-4b08-83b8-46fdb7fee0bb\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " Oct 29 23:02:41.341936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.341891 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8\" (UniqueName: \"kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8\") pod \"b144f021-128f-4b08-83b8-46fdb7fee0bb\" (UID: \"b144f021-128f-4b08-83b8-46fdb7fee0bb\") " Oct 29 23:02:41.342015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.341999 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z\" (UniqueName: \"kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z\") pod \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\" (UID: \"6a9c8e10-9a9e-497e-85fc-e0b241c25f60\") " Oct 29 23:02:41.342321 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.342276 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume" pod "b144f021-128f-4b08-83b8-46fdb7fee0bb" (UID: "b144f021-128f-4b08-83b8-46fdb7fee0bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:41.342805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.342784 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b144f021-128f-4b08-83b8-46fdb7fee0bb-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:41.343854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.343832 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume" pod "6a9c8e10-9a9e-497e-85fc-e0b241c25f60" (UID: "6a9c8e10-9a9e-497e-85fc-e0b241c25f60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:41.345342 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.345314 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8" pod "b144f021-128f-4b08-83b8-46fdb7fee0bb" (UID: "b144f021-128f-4b08-83b8-46fdb7fee0bb"). InnerVolumeSpecName "kube-api-access-dpts8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:41.345504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.345486 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z" pod "6a9c8e10-9a9e-497e-85fc-e0b241c25f60" (UID: "6a9c8e10-9a9e-497e-85fc-e0b241c25f60"). InnerVolumeSpecName "kube-api-access-5pd7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:41.443658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.443625 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:41.443658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.443655 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpts8\" (UniqueName: \"kubernetes.io/projected/b144f021-128f-4b08-83b8-46fdb7fee0bb-kube-api-access-dpts8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:41.443658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.443665 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pd7z\" (UniqueName: \"kubernetes.io/projected/6a9c8e10-9a9e-497e-85fc-e0b241c25f60-kube-api-access-5pd7z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:41.688037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.687993 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7"] Oct 29 23:02:41.715881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.714909 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-799768d694-x5lm7"] Oct 29 23:02:41.739898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.739858 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd"] Oct 29 23:02:41.743988 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:41.743960 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-5856fcbb5f-f54wd"] Oct 29 23:02:42.556362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:42.556325 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:42.557049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:42.557016 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:43.224865 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:43.224825 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9c8e10-9a9e-497e-85fc-e0b241c25f60" path="/var/lib/kubelet/pods/6a9c8e10-9a9e-497e-85fc-e0b241c25f60/volumes" Oct 29 23:02:43.225441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:43.225421 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b144f021-128f-4b08-83b8-46fdb7fee0bb" path="/var/lib/kubelet/pods/b144f021-128f-4b08-83b8-46fdb7fee0bb/volumes" Oct 29 23:02:44.573609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:44.573580 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf"] Oct 29 23:02:44.574548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:44.573859 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" containerID="cri-o://660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85" gracePeriod=30 Oct 29 23:02:45.226604 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.226578 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt"] Oct 29 23:02:45.227206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227191 5367 cpu_manager.go:414] "containerMap: removing container" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" Oct 29 23:02:45.227258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227207 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" Oct 29 23:02:45.227258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227227 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1421c097-e373-4efa-9b49-256eb3438c6a" containerName="revoke-keys" Oct 29 23:02:45.227258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227232 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1421c097-e373-4efa-9b49-256eb3438c6a" containerName="revoke-keys" Oct 29 23:02:45.227258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227243 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b144f021-128f-4b08-83b8-46fdb7fee0bb" containerName="payload-processing" Oct 29 23:02:45.227258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227248 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b144f021-128f-4b08-83b8-46fdb7fee0bb" containerName="payload-processing" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227260 5367 cpu_manager.go:414] "containerMap: removing container" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227265 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227272 5367 cpu_manager.go:414] "containerMap: removing container" podUID="294ad7f3-5730-49a7-85d0-19ff1b8b13b8" containerName="revoke-keys" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227277 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294ad7f3-5730-49a7-85d0-19ff1b8b13b8" containerName="revoke-keys" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227288 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227293 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227301 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6a9c8e10-9a9e-497e-85fc-e0b241c25f60" containerName="payload-pre-processing" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227306 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9c8e10-9a9e-497e-85fc-e0b241c25f60" containerName="payload-pre-processing" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227400 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="378e0f62-b817-47b8-ade5-40585173dc5d" containerName="main" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227409 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1421c097-e373-4efa-9b49-256eb3438c6a" containerName="revoke-keys" Oct 29 23:02:45.227417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227416 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b144f021-128f-4b08-83b8-46fdb7fee0bb" containerName="payload-processing" Oct 29 23:02:45.227826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227424 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb0eb6c-2930-431f-a36d-c6f3dda9b228" containerName="main" Oct 29 23:02:45.227826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227429 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="294ad7f3-5730-49a7-85d0-19ff1b8b13b8" containerName="revoke-keys" Oct 29 23:02:45.227826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227437 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c9e995-c1bf-4f07-ac73-73248472db5d" containerName="main" Oct 29 23:02:45.227826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.227442 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9c8e10-9a9e-497e-85fc-e0b241c25f60" containerName="payload-pre-processing" Oct 29 23:02:45.232376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.232357 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.252166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.252132 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt"] Oct 29 23:02:45.317941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.317908 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:45.318183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.318164 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:02:45.326936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.326909 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.327173 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.327150 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.327258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.327242 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkwn\" (UniqueName: \"kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.429220 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.429187 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.429402 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.429313 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.429402 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.429369 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkwn\" (UniqueName: \"kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.430007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.429985 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.431850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.431830 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.450650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.450621 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkwn\" (UniqueName: \"kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.549349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.549278 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:45.859581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:45.859543 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt"] Oct 29 23:02:45.859984 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:02:45.859935 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31f7468_f5fe_4dbe_bf70_8956854b4c3e.slice/crio-6a2e510e41ad9e3d0fab24cb9aa86ec467b45f84ce3f083b2da60d42fc5f4893 WatchSource:0}: Error finding container 6a2e510e41ad9e3d0fab24cb9aa86ec467b45f84ce3f083b2da60d42fc5f4893: Status 404 returned error can't find the container with id 6a2e510e41ad9e3d0fab24cb9aa86ec467b45f84ce3f083b2da60d42fc5f4893 Oct 29 23:02:46.395242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:46.395199 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" event={"ID":"a31f7468-f5fe-4dbe-bf70-8956854b4c3e","Type":"ContainerStarted","Data":"8ddb8fab79e09388eea322b30b91ebad99315a30886bb7907e398dd87664056c"} Oct 29 23:02:46.395418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:46.395268 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" event={"ID":"a31f7468-f5fe-4dbe-bf70-8956854b4c3e","Type":"ContainerStarted","Data":"6a2e510e41ad9e3d0fab24cb9aa86ec467b45f84ce3f083b2da60d42fc5f4893"} Oct 29 23:02:46.438083 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:46.437985 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" podStartSLOduration=1.437970685 podStartE2EDuration="1.437970685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:46.435464856 +0000 UTC m=+2833.329539777" watchObservedRunningTime="2026-10-29 23:02:46.437970685 +0000 UTC m=+2833.332045645" Oct 29 23:02:47.405061 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:47.405029 5367 generic.go:356] "Generic (PLEG): container finished" podID="a31f7468-f5fe-4dbe-bf70-8956854b4c3e" containerID="8ddb8fab79e09388eea322b30b91ebad99315a30886bb7907e398dd87664056c" exitCode=0 Oct 29 23:02:47.405459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:47.405151 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" event={"ID":"a31f7468-f5fe-4dbe-bf70-8956854b4c3e","Type":"ContainerDied","Data":"8ddb8fab79e09388eea322b30b91ebad99315a30886bb7907e398dd87664056c"} Oct 29 23:02:48.718180 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.718155 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:48.903192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.903152 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn\" (UniqueName: \"kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn\") pod \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " Oct 29 23:02:48.903386 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.903310 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca\") pod \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " Oct 29 23:02:48.903462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.903449 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca\") pod \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\" (UID: \"a31f7468-f5fe-4dbe-bf70-8956854b4c3e\") " Oct 29 23:02:48.903724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.903660 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca" pod "a31f7468-f5fe-4dbe-bf70-8956854b4c3e" (UID: "a31f7468-f5fe-4dbe-bf70-8956854b4c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:48.904080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.904056 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:48.905320 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.905293 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn" pod "a31f7468-f5fe-4dbe-bf70-8956854b4c3e" (UID: "a31f7468-f5fe-4dbe-bf70-8956854b4c3e"). InnerVolumeSpecName "kube-api-access-2zkwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:48.905420 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:48.905347 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca" pod "a31f7468-f5fe-4dbe-bf70-8956854b4c3e" (UID: "a31f7468-f5fe-4dbe-bf70-8956854b4c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:49.005791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.005765 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.005791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.005790 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zkwn\" (UniqueName: \"kubernetes.io/projected/a31f7468-f5fe-4dbe-bf70-8956854b4c3e-kube-api-access-2zkwn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.500801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.500691 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" event={"ID":"a31f7468-f5fe-4dbe-bf70-8956854b4c3e","Type":"ContainerDied","Data":"6a2e510e41ad9e3d0fab24cb9aa86ec467b45f84ce3f083b2da60d42fc5f4893"} Oct 29 23:02:49.500801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.500794 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a2e510e41ad9e3d0fab24cb9aa86ec467b45f84ce3f083b2da60d42fc5f4893" Oct 29 23:02:49.501001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.500692 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt" Oct 29 23:02:49.881820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.881795 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 23:02:49.915972 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.915941 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.915978 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916004 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916053 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916150 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916183 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916249 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916281 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916365 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw\") pod \"1e462c93-e10b-4572-b36f-b934c912f870\" (UID: \"1e462c93-e10b-4572-b36f-b934c912f870\") " Oct 29 23:02:49.916637 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916456 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.916637 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916473 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:49.916784 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916683 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.916849 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916831 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.917013 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916979 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.917013 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.916999 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.917140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.917025 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.917140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.917043 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.917140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.917056 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e462c93-e10b-4572-b36f-b934c912f870-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.918548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.918524 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:49.918647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.918626 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "kube-api-access-6dctw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:49.918792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.918773 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:49.918865 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:49.918788 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy" pod "1e462c93-e10b-4572-b36f-b934c912f870" (UID: "1e462c93-e10b-4572-b36f-b934c912f870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.019039 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.019000 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.019039 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.019042 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dctw\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-kube-api-access-6dctw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.019245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.019056 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e462c93-e10b-4572-b36f-b934c912f870-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.019245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.019071 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e462c93-e10b-4572-b36f-b934c912f870-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.019245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.019083 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e462c93-e10b-4572-b36f-b934c912f870-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.089765 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.089726 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr"] Oct 29 23:02:50.510400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.510369 5367 generic.go:356] "Generic (PLEG): container finished" podID="1e462c93-e10b-4572-b36f-b934c912f870" containerID="660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85" exitCode=0 Oct 29 23:02:50.510606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.510429 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" Oct 29 23:02:50.510606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.510460 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" event={"ID":"1e462c93-e10b-4572-b36f-b934c912f870","Type":"ContainerDied","Data":"660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85"} Oct 29 23:02:50.510606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.510526 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf" event={"ID":"1e462c93-e10b-4572-b36f-b934c912f870","Type":"ContainerDied","Data":"e0f57bcf2a62c79b11ba1c0e850297a707852c647f0c3ee0a1da42130a8a6f0e"} Oct 29 23:02:50.510606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.510559 5367 scope.go:122] "RemoveContainer" containerID="660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85" Oct 29 23:02:50.522571 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.522545 5367 scope.go:122] "RemoveContainer" containerID="484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136" Oct 29 23:02:50.532418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.532399 5367 scope.go:122] "RemoveContainer" containerID="660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85" Oct 29 23:02:50.532733 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:50.532714 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85\": container with ID starting with 660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85 not found: ID does not exist" containerID="660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85" Oct 29 23:02:50.532820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.532744 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85"} err="failed to get container status \"660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85\": rpc error: code = NotFound desc = could not find container \"660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85\": container with ID starting with 660e4ec7c1b477cc94f2afe9a79d293cccd8a1cb979e3ee5e5400766d3176c85 not found: ID does not exist" Oct 29 23:02:50.532820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.532762 5367 scope.go:122] "RemoveContainer" containerID="484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136" Oct 29 23:02:50.533051 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:50.533035 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136\": container with ID starting with 484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136 not found: ID does not exist" containerID="484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136" Oct 29 23:02:50.533098 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.533056 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136"} err="failed to get container status \"484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136\": rpc error: code = NotFound desc = could not find container \"484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136\": container with ID starting with 484c81539324a1e8b804fd26ce8d4e68607d4ed962c992fcbb6f9f8072145136 not found: ID does not exist" Oct 29 23:02:50.574397 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.574365 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf"] Oct 29 23:02:50.585193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.585162 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-k5dcf"] Oct 29 23:02:50.899095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.899071 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 23:02:50.935580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.935551 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume\") pod \"b474646f-cd1b-4689-a935-46b6468ab79c\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " Oct 29 23:02:50.935741 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.935608 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj\" (UniqueName: \"kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj\") pod \"b474646f-cd1b-4689-a935-46b6468ab79c\" (UID: \"b474646f-cd1b-4689-a935-46b6468ab79c\") " Oct 29 23:02:50.935948 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.935925 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume" pod "b474646f-cd1b-4689-a935-46b6468ab79c" (UID: "b474646f-cd1b-4689-a935-46b6468ab79c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:50.936269 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.936248 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b474646f-cd1b-4689-a935-46b6468ab79c-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.937826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.937799 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj" pod "b474646f-cd1b-4689-a935-46b6468ab79c" (UID: "b474646f-cd1b-4689-a935-46b6468ab79c"). InnerVolumeSpecName "kube-api-access-hfqmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:50.941470 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:50.941448 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 23:02:51.036809 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.036725 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9\" (UniqueName: \"kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9\") pod \"cd0b05f3-9543-4a70-ac49-175571ea9adf\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " Oct 29 23:02:51.036992 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.036890 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume\") pod \"cd0b05f3-9543-4a70-ac49-175571ea9adf\" (UID: \"cd0b05f3-9543-4a70-ac49-175571ea9adf\") " Oct 29 23:02:51.037251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.037228 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume" pod "cd0b05f3-9543-4a70-ac49-175571ea9adf" (UID: "cd0b05f3-9543-4a70-ac49-175571ea9adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:51.037331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.037300 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqmj\" (UniqueName: \"kubernetes.io/projected/b474646f-cd1b-4689-a935-46b6468ab79c-kube-api-access-hfqmj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:51.039000 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.038980 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9" pod "cd0b05f3-9543-4a70-ac49-175571ea9adf" (UID: "cd0b05f3-9543-4a70-ac49-175571ea9adf"). InnerVolumeSpecName "kube-api-access-7wng9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:51.139143 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.139107 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b05f3-9543-4a70-ac49-175571ea9adf-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:51.139143 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.139137 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wng9\" (UniqueName: \"kubernetes.io/projected/cd0b05f3-9543-4a70-ac49-175571ea9adf-kube-api-access-7wng9\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:02:51.222640 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.222606 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e462c93-e10b-4572-b36f-b934c912f870" path="/var/lib/kubelet/pods/1e462c93-e10b-4572-b36f-b934c912f870/volumes" Oct 29 23:02:51.522284 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.522247 5367 generic.go:356] "Generic (PLEG): container finished" podID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerID="a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f" exitCode=0 Oct 29 23:02:51.522515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.522292 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" Oct 29 23:02:51.522515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.522338 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" event={"ID":"cd0b05f3-9543-4a70-ac49-175571ea9adf","Type":"ContainerDied","Data":"a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f"} Oct 29 23:02:51.522515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.522393 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj" event={"ID":"cd0b05f3-9543-4a70-ac49-175571ea9adf","Type":"ContainerDied","Data":"072f3897c2c412212968cf70f00cc7358293d6e533ee830dfc740e683df72b7e"} Oct 29 23:02:51.522515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.522425 5367 scope.go:122] "RemoveContainer" containerID="a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f" Oct 29 23:02:51.524138 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.524117 5367 generic.go:356] "Generic (PLEG): container finished" podID="b474646f-cd1b-4689-a935-46b6468ab79c" containerID="8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1" exitCode=0 Oct 29 23:02:51.524256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.524161 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" Oct 29 23:02:51.524256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.524187 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" event={"ID":"b474646f-cd1b-4689-a935-46b6468ab79c","Type":"ContainerDied","Data":"8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1"} Oct 29 23:02:51.524256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.524221 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf" event={"ID":"b474646f-cd1b-4689-a935-46b6468ab79c","Type":"ContainerDied","Data":"cc9f4311ed8748537bb27659d22a1dfcf0055bb080152c17eed34292f91b09c4"} Oct 29 23:02:51.534240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.534210 5367 scope.go:122] "RemoveContainer" containerID="a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f" Oct 29 23:02:51.534536 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:51.534512 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f\": container with ID starting with a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f not found: ID does not exist" containerID="a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f" Oct 29 23:02:51.534603 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.534543 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f"} err="failed to get container status \"a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f\": rpc error: code = NotFound desc = could not find container \"a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f\": container with ID starting with a6e5de4e8d0a95b859823221303fc35b8d2da4a2bb6e25988730066ada5c311f not found: ID does not exist" Oct 29 23:02:51.534603 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.534560 5367 scope.go:122] "RemoveContainer" containerID="8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1" Oct 29 23:02:51.545793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.545775 5367 scope.go:122] "RemoveContainer" containerID="8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1" Oct 29 23:02:51.546170 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:51.546146 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1\": container with ID starting with 8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1 not found: ID does not exist" containerID="8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1" Oct 29 23:02:51.546281 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.546180 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1"} err="failed to get container status \"8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1\": rpc error: code = NotFound desc = could not find container \"8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1\": container with ID starting with 8eb0977de8c8d6c73da2f43ff150f26d6961c17c283b6d76f6308d1bd1a229e1 not found: ID does not exist" Oct 29 23:02:51.577605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.577568 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf"] Oct 29 23:02:51.581990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.581965 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-879596-7f976c6496-prjrf"] Oct 29 23:02:51.612654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.612614 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj"] Oct 29 23:02:51.616456 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:51.616430 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-879596-5b795898bc-hgvxj"] Oct 29 23:02:52.331585 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:02:52.331552 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:02:53.227785 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:53.227741 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" path="/var/lib/kubelet/pods/b474646f-cd1b-4689-a935-46b6468ab79c/volumes" Oct 29 23:02:53.233264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:53.233228 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" path="/var/lib/kubelet/pods/cd0b05f3-9543-4a70-ac49-175571ea9adf/volumes" Oct 29 23:02:55.687243 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:02:55.687209 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s"] Oct 29 23:03:00.688098 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:00.688060 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerName="payload-processing" containerID="cri-o://5690883cd09be426793c228bb22c289dc660df6e26155fcba22de6a7f146ecd6" gracePeriod=25 Oct 29 23:03:01.171821 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.171788 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7"] Oct 29 23:03:01.172393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172379 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a31f7468-f5fe-4dbe-bf70-8956854b4c3e" containerName="revoke-keys" Oct 29 23:03:01.172450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172395 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31f7468-f5fe-4dbe-bf70-8956854b4c3e" containerName="revoke-keys" Oct 29 23:03:01.172450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172423 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" Oct 29 23:03:01.172450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172428 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" Oct 29 23:03:01.172450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172440 5367 cpu_manager.go:414] "containerMap: removing container" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerName="payload-pre-processing" Oct 29 23:03:01.172450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172446 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerName="payload-pre-processing" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172461 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" containerName="payload-processing" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172466 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" containerName="payload-processing" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172476 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172481 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172562 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b474646f-cd1b-4689-a935-46b6468ab79c" containerName="payload-processing" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172573 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172581 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31f7468-f5fe-4dbe-bf70-8956854b4c3e" containerName="revoke-keys" Oct 29 23:03:01.172614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.172593 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0b05f3-9543-4a70-ac49-175571ea9adf" containerName="payload-pre-processing" Oct 29 23:03:01.177741 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.177724 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.194925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.194897 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7"] Oct 29 23:03:01.201528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.201491 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:01.201662 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.201555 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:03:01.246831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.246805 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.246982 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.246838 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.247057 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.247045 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bv8g\" (UniqueName: \"kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.348155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.348116 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bv8g\" (UniqueName: \"kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.348332 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.348219 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.348332 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.348252 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.349003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.348981 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.350599 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.350580 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.368996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.368971 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bv8g\" (UniqueName: \"kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.491595 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.491513 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:01.525743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.525689 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt"] Oct 29 23:03:01.526447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.526431 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e462c93-e10b-4572-b36f-b934c912f870" containerName="istio-proxy" Oct 29 23:03:01.531295 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.531269 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.538406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.538374 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh"] Oct 29 23:03:01.543762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.543633 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.556719 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.555405 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt"] Oct 29 23:03:01.566584 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.564881 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh"] Oct 29 23:03:01.652443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.652409 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7r28\" (UniqueName: \"kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.652607 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.652497 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.652607 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.652574 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fqf\" (UniqueName: \"kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.652767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.652670 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.652767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.652726 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.652859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.652806 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.754672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.754578 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.754672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.754618 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.754672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.754664 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.755356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.754800 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7r28\" (UniqueName: \"kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.755356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.754874 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.755356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.754892 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fqf\" (UniqueName: \"kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.755518 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.755496 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.755556 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.755499 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.757644 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.757617 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.757644 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.757631 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.784424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.784393 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fqf\" (UniqueName: \"kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf\") pod \"maas-api-revoke-sub-dafc37b4-dvsmh\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:01.785004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.784984 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7r28\" (UniqueName: \"kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28\") pod \"maas-api-revoke-sub-44907e0a-mxwbt\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.816205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.816177 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7"] Oct 29 23:03:01.816548 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:01.816523 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa13ab7_8f20_4ee1_9b80_e74a428adaaf.slice/crio-1bee7848a23126da49416e8f2c31fd9fdf450d9d71f223546e0ce9b5eb39138b WatchSource:0}: Error finding container 1bee7848a23126da49416e8f2c31fd9fdf450d9d71f223546e0ce9b5eb39138b: Status 404 returned error can't find the container with id 1bee7848a23126da49416e8f2c31fd9fdf450d9d71f223546e0ce9b5eb39138b Oct 29 23:03:01.859546 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.859519 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:01.891471 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:01.891439 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:02.032537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.030884 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk"] Oct 29 23:03:02.039494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.037408 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b615a1bc-rrgzk"] Oct 29 23:03:02.164884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.164854 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt"] Oct 29 23:03:02.168227 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:02.168203 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd653f73e_726c_475d_b931_367ae7dfa5d8.slice/crio-4fc2d966a3b653d7d7114dfc8f7ba1abd3fb287f898d6cc355be6934b0062409 WatchSource:0}: Error finding container 4fc2d966a3b653d7d7114dfc8f7ba1abd3fb287f898d6cc355be6934b0062409: Status 404 returned error can't find the container with id 4fc2d966a3b653d7d7114dfc8f7ba1abd3fb287f898d6cc355be6934b0062409 Oct 29 23:03:02.264479 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:02.264448 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a18a25e_9a4a_4f3e_aeaf_eb79c0cdecfc.slice/crio-9eb3ebe2b1423af65e4f46a68d2a787cfff17adf47ad76a5a289eeef58c773d9 WatchSource:0}: Error finding container 9eb3ebe2b1423af65e4f46a68d2a787cfff17adf47ad76a5a289eeef58c773d9: Status 404 returned error can't find the container with id 9eb3ebe2b1423af65e4f46a68d2a787cfff17adf47ad76a5a289eeef58c773d9 Oct 29 23:03:02.265975 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.265950 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh"] Oct 29 23:03:02.704862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.704825 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" event={"ID":"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc","Type":"ContainerStarted","Data":"8716665b5ec812c6614454d278ab3c390bb74b5879cdcdb4db109627d6dbaabe"} Oct 29 23:03:02.705040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.704882 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" event={"ID":"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc","Type":"ContainerStarted","Data":"9eb3ebe2b1423af65e4f46a68d2a787cfff17adf47ad76a5a289eeef58c773d9"} Oct 29 23:03:02.706603 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.706575 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" event={"ID":"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf","Type":"ContainerStarted","Data":"e3d4de5f3314599033b635708fba17c8b775524acf777fe2219963bf6687ddd7"} Oct 29 23:03:02.706783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.706627 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" event={"ID":"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf","Type":"ContainerStarted","Data":"1bee7848a23126da49416e8f2c31fd9fdf450d9d71f223546e0ce9b5eb39138b"} Oct 29 23:03:02.708198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.708176 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" event={"ID":"d653f73e-726c-475d-b931-367ae7dfa5d8","Type":"ContainerStarted","Data":"66ab1e32e0ed50301a9b8eaf423dd716a97608aefcfa1a4b969e1809257d9e68"} Oct 29 23:03:02.708289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.708214 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" event={"ID":"d653f73e-726c-475d-b931-367ae7dfa5d8","Type":"ContainerStarted","Data":"4fc2d966a3b653d7d7114dfc8f7ba1abd3fb287f898d6cc355be6934b0062409"} Oct 29 23:03:02.745459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.745413 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" podStartSLOduration=1.745396679 podStartE2EDuration="1.745396679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:02.743814466 +0000 UTC m=+2849.637889384" watchObservedRunningTime="2026-10-29 23:03:02.745396679 +0000 UTC m=+2849.639471605" Oct 29 23:03:02.784882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.784829 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" podStartSLOduration=1.78481585 podStartE2EDuration="1.78481585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:02.781587864 +0000 UTC m=+2849.675662789" watchObservedRunningTime="2026-10-29 23:03:02.78481585 +0000 UTC m=+2849.678890771" Oct 29 23:03:02.820907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:02.820851 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" podStartSLOduration=1.820831347 podStartE2EDuration="1.820831347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:02.819217636 +0000 UTC m=+2849.713292559" watchObservedRunningTime="2026-10-29 23:03:02.820831347 +0000 UTC m=+2849.714906272" Oct 29 23:03:03.226157 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.226120 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24914c40-9563-4ca3-9332-cf50abf3a3b0" path="/var/lib/kubelet/pods/24914c40-9563-4ca3-9332-cf50abf3a3b0/volumes" Oct 29 23:03:03.717481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.717448 5367 generic.go:356] "Generic (PLEG): container finished" podID="7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" containerID="8716665b5ec812c6614454d278ab3c390bb74b5879cdcdb4db109627d6dbaabe" exitCode=0 Oct 29 23:03:03.717689 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.717540 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" event={"ID":"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc","Type":"ContainerDied","Data":"8716665b5ec812c6614454d278ab3c390bb74b5879cdcdb4db109627d6dbaabe"} Oct 29 23:03:03.719112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.719088 5367 generic.go:356] "Generic (PLEG): container finished" podID="2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" containerID="e3d4de5f3314599033b635708fba17c8b775524acf777fe2219963bf6687ddd7" exitCode=0 Oct 29 23:03:03.719206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.719152 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" event={"ID":"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf","Type":"ContainerDied","Data":"e3d4de5f3314599033b635708fba17c8b775524acf777fe2219963bf6687ddd7"} Oct 29 23:03:03.720771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.720747 5367 generic.go:356] "Generic (PLEG): container finished" podID="d653f73e-726c-475d-b931-367ae7dfa5d8" containerID="66ab1e32e0ed50301a9b8eaf423dd716a97608aefcfa1a4b969e1809257d9e68" exitCode=0 Oct 29 23:03:03.720866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:03.720774 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" event={"ID":"d653f73e-726c-475d-b931-367ae7dfa5d8","Type":"ContainerDied","Data":"66ab1e32e0ed50301a9b8eaf423dd716a97608aefcfa1a4b969e1809257d9e68"} Oct 29 23:03:04.898424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:04.898385 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:04.968624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:04.968596 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:04.991510 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:04.991455 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:05.018302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.016485 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca\") pod \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " Oct 29 23:03:05.018302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.016770 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca\") pod \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " Oct 29 23:03:05.018302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.016842 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g\" (UniqueName: \"kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g\") pod \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\" (UID: \"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf\") " Oct 29 23:03:05.027845 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.027790 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca" pod "2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" (UID: "2fa13ab7-8f20-4ee1-9b80-e74a428adaaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:05.041221 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.034287 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca" pod "2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" (UID: "2fa13ab7-8f20-4ee1-9b80-e74a428adaaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.093533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.093475 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g" pod "2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" (UID: "2fa13ab7-8f20-4ee1-9b80-e74a428adaaf"). InnerVolumeSpecName "kube-api-access-4bv8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.111792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.111736 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" containerID="cri-o://8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869" gracePeriod=45 Oct 29 23:03:05.118119 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118095 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28\" (UniqueName: \"kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28\") pod \"d653f73e-726c-475d-b931-367ae7dfa5d8\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " Oct 29 23:03:05.118251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118210 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca\") pod \"d653f73e-726c-475d-b931-367ae7dfa5d8\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " Oct 29 23:03:05.118251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118244 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca\") pod \"d653f73e-726c-475d-b931-367ae7dfa5d8\" (UID: \"d653f73e-726c-475d-b931-367ae7dfa5d8\") " Oct 29 23:03:05.118369 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118300 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca\") pod \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " Oct 29 23:03:05.118445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118415 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf\" (UniqueName: \"kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf\") pod \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " Oct 29 23:03:05.118575 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118557 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca\") pod \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\" (UID: \"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc\") " Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118679 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca" pod "d653f73e-726c-475d-b931-367ae7dfa5d8" (UID: "d653f73e-726c-475d-b931-367ae7dfa5d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.118965 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca" pod "7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" (UID: "7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.119132 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.119148 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.119157 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.119167 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.119249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.119176 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bv8g\" (UniqueName: \"kubernetes.io/projected/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf-kube-api-access-4bv8g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.120841 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.120817 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca" pod "d653f73e-726c-475d-b931-367ae7dfa5d8" (UID: "d653f73e-726c-475d-b931-367ae7dfa5d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.120963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.120853 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca" pod "7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" (UID: "7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.120963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.120910 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28" pod "d653f73e-726c-475d-b931-367ae7dfa5d8" (UID: "d653f73e-726c-475d-b931-367ae7dfa5d8"). InnerVolumeSpecName "kube-api-access-p7r28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.122032 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.122013 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf" pod "7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" (UID: "7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc"). InnerVolumeSpecName "kube-api-access-q2fqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.220560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.220529 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d653f73e-726c-475d-b931-367ae7dfa5d8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.220560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.220557 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.220560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.220566 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2fqf\" (UniqueName: \"kubernetes.io/projected/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc-kube-api-access-q2fqf\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.220879 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.220576 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7r28\" (UniqueName: \"kubernetes.io/projected/d653f73e-726c-475d-b931-367ae7dfa5d8-kube-api-access-p7r28\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.376121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.376096 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 23:03:05.524812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.524778 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir\") pod \"c34e200d-12be-48ab-917a-b2ab3a05a55a\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " Oct 29 23:03:05.525043 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.524979 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm\") pod \"c34e200d-12be-48ab-917a-b2ab3a05a55a\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " Oct 29 23:03:05.525043 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525020 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache\") pod \"c34e200d-12be-48ab-917a-b2ab3a05a55a\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " Oct 29 23:03:05.525200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525058 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw\" (UniqueName: \"kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw\") pod \"c34e200d-12be-48ab-917a-b2ab3a05a55a\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " Oct 29 23:03:05.525200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525087 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir" pod "c34e200d-12be-48ab-917a-b2ab3a05a55a" (UID: "c34e200d-12be-48ab-917a-b2ab3a05a55a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.525200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525095 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home\") pod \"c34e200d-12be-48ab-917a-b2ab3a05a55a\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " Oct 29 23:03:05.525443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525323 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache" pod "c34e200d-12be-48ab-917a-b2ab3a05a55a" (UID: "c34e200d-12be-48ab-917a-b2ab3a05a55a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.525443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525344 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home" pod "c34e200d-12be-48ab-917a-b2ab3a05a55a" (UID: "c34e200d-12be-48ab-917a-b2ab3a05a55a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.525443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525348 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs\") pod \"c34e200d-12be-48ab-917a-b2ab3a05a55a\" (UID: \"c34e200d-12be-48ab-917a-b2ab3a05a55a\") " Oct 29 23:03:05.526019 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.525996 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.526128 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.526020 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.526128 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.526034 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.527264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.527244 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm" pod "c34e200d-12be-48ab-917a-b2ab3a05a55a" (UID: "c34e200d-12be-48ab-917a-b2ab3a05a55a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.527336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.527316 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs" pod "c34e200d-12be-48ab-917a-b2ab3a05a55a" (UID: "c34e200d-12be-48ab-917a-b2ab3a05a55a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.527583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.527557 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw" pod "c34e200d-12be-48ab-917a-b2ab3a05a55a" (UID: "c34e200d-12be-48ab-917a-b2ab3a05a55a"). InnerVolumeSpecName "kube-api-access-2ckpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.580678 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.580647 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd"] Oct 29 23:03:05.581319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581303 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" Oct 29 23:03:05.581367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581320 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" Oct 29 23:03:05.581367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581339 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" containerName="revoke-keys" Oct 29 23:03:05.581367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581345 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581374 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581380 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581389 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d653f73e-726c-475d-b931-367ae7dfa5d8" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581393 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d653f73e-726c-475d-b931-367ae7dfa5d8" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581478 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581491 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581500 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d653f73e-726c-475d-b931-367ae7dfa5d8" containerName="revoke-keys" Oct 29 23:03:05.581523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.581508 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerName="main" Oct 29 23:03:05.590199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.590176 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.606201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.606173 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd"] Oct 29 23:03:05.629808 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.629752 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34e200d-12be-48ab-917a-b2ab3a05a55a-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.630742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.630049 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckpw\" (UniqueName: \"kubernetes.io/projected/c34e200d-12be-48ab-917a-b2ab3a05a55a-kube-api-access-2ckpw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.631478 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.630567 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34e200d-12be-48ab-917a-b2ab3a05a55a-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.732335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.732299 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.732529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.732507 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.732672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.732651 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n8h\" (UniqueName: \"kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.745188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.745154 5367 generic.go:356] "Generic (PLEG): container finished" podID="c34e200d-12be-48ab-917a-b2ab3a05a55a" containerID="8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869" exitCode=0 Oct 29 23:03:05.745362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.745196 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" Oct 29 23:03:05.745362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.745235 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" event={"ID":"c34e200d-12be-48ab-917a-b2ab3a05a55a","Type":"ContainerDied","Data":"8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869"} Oct 29 23:03:05.745362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.745277 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr" event={"ID":"c34e200d-12be-48ab-917a-b2ab3a05a55a","Type":"ContainerDied","Data":"e91883b72e55ce0b1f85b3e86a0eec91e12d1c91287ff294cb44e80f08c71d6a"} Oct 29 23:03:05.745362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.745300 5367 scope.go:122] "RemoveContainer" containerID="8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869" Oct 29 23:03:05.747311 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.747291 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" Oct 29 23:03:05.747413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.747304 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh" event={"ID":"7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc","Type":"ContainerDied","Data":"9eb3ebe2b1423af65e4f46a68d2a787cfff17adf47ad76a5a289eeef58c773d9"} Oct 29 23:03:05.747413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.747351 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb3ebe2b1423af65e4f46a68d2a787cfff17adf47ad76a5a289eeef58c773d9" Oct 29 23:03:05.749377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.749357 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" Oct 29 23:03:05.749505 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.749377 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7" event={"ID":"2fa13ab7-8f20-4ee1-9b80-e74a428adaaf","Type":"ContainerDied","Data":"1bee7848a23126da49416e8f2c31fd9fdf450d9d71f223546e0ce9b5eb39138b"} Oct 29 23:03:05.749505 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.749420 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bee7848a23126da49416e8f2c31fd9fdf450d9d71f223546e0ce9b5eb39138b" Oct 29 23:03:05.751244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.751223 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" event={"ID":"d653f73e-726c-475d-b931-367ae7dfa5d8","Type":"ContainerDied","Data":"4fc2d966a3b653d7d7114dfc8f7ba1abd3fb287f898d6cc355be6934b0062409"} Oct 29 23:03:05.751347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.751263 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc2d966a3b653d7d7114dfc8f7ba1abd3fb287f898d6cc355be6934b0062409" Oct 29 23:03:05.751347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.751230 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt" Oct 29 23:03:05.758308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.758292 5367 scope.go:122] "RemoveContainer" containerID="8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869" Oct 29 23:03:05.758616 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:05.758589 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869\": container with ID starting with 8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869 not found: ID does not exist" containerID="8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869" Oct 29 23:03:05.758685 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.758629 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869"} err="failed to get container status \"8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869\": rpc error: code = NotFound desc = could not find container \"8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869\": container with ID starting with 8142182e740c20d76bf54439e38535aedeeab0a0dfd786f181db02b2db4d6869 not found: ID does not exist" Oct 29 23:03:05.833794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.833710 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.833971 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.833812 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.833971 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.833880 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n8h\" (UniqueName: \"kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.834416 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.834390 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.836223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.836197 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.851441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.851415 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr"] Oct 29 23:03:05.853439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.853412 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n8h\" (UniqueName: \"kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h\") pod \"maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:05.858424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.858395 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/simulator-w2-8d3632-kserve-5db55fb7d5-cx5gr"] Oct 29 23:03:05.906507 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:05.906480 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:06.211885 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:06.211854 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f18f91_7ec1_4f06_9f23_ca0e3365d4c8.slice/crio-eb16777d8a0e9c1619378f7f740a1e06f24d2af0f4a09de91aed1d6aaf08af14 WatchSource:0}: Error finding container eb16777d8a0e9c1619378f7f740a1e06f24d2af0f4a09de91aed1d6aaf08af14: Status 404 returned error can't find the container with id eb16777d8a0e9c1619378f7f740a1e06f24d2af0f4a09de91aed1d6aaf08af14 Oct 29 23:03:06.212032 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:06.211906 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd"] Oct 29 23:03:06.762073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:06.761986 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" event={"ID":"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8","Type":"ContainerStarted","Data":"797951fe3f57fd5bca8f37ececa432f3c701dc761a5cb36abd5f3d61a31f9b5d"} Oct 29 23:03:06.762073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:06.762042 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" event={"ID":"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8","Type":"ContainerStarted","Data":"eb16777d8a0e9c1619378f7f740a1e06f24d2af0f4a09de91aed1d6aaf08af14"} Oct 29 23:03:06.802184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:06.802132 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" podStartSLOduration=1.8021174229999999 podStartE2EDuration="1.802117423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:06.800805916 +0000 UTC m=+2853.694880837" watchObservedRunningTime="2026-10-29 23:03:06.802117423 +0000 UTC m=+2853.696192343" Oct 29 23:03:07.221897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:07.221866 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34e200d-12be-48ab-917a-b2ab3a05a55a" path="/var/lib/kubelet/pods/c34e200d-12be-48ab-917a-b2ab3a05a55a/volumes" Oct 29 23:03:07.487644 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:07.487555 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:03:07.772892 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:07.772808 5367 generic.go:356] "Generic (PLEG): container finished" podID="73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" containerID="797951fe3f57fd5bca8f37ececa432f3c701dc761a5cb36abd5f3d61a31f9b5d" exitCode=0 Oct 29 23:03:07.773042 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:07.772888 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" event={"ID":"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8","Type":"ContainerDied","Data":"797951fe3f57fd5bca8f37ececa432f3c701dc761a5cb36abd5f3d61a31f9b5d"} Oct 29 23:03:08.321105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.321069 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b"] Oct 29 23:03:08.325675 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.325657 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.340959 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.340915 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3e2b2c\"" Oct 29 23:03:08.345483 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.345459 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b"] Oct 29 23:03:08.352685 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.352660 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3e2b2c-dockercfg-2t6kn\"" Oct 29 23:03:08.472716 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.472675 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.472716 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.472720 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tkmh\" (UniqueName: \"kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh\") pod \"payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.574169 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.574083 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.574169 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.574132 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tkmh\" (UniqueName: \"kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh\") pod \"payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.574904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.574878 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.595018 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.594989 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tkmh\" (UniqueName: \"kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh\") pod \"payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:08.640075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:08.640048 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:09.044407 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.026671 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b"] Oct 29 23:03:09.210270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.210243 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:09.292543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.292449 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn"] Oct 29 23:03:09.298448 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.298409 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-879596-nqrpn"] Oct 29 23:03:09.327233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.323162 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca\") pod \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " Oct 29 23:03:09.327233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.325315 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h\" (UniqueName: \"kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h\") pod \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " Oct 29 23:03:09.327233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.325499 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca\") pod \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\" (UID: \"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8\") " Oct 29 23:03:09.327233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.326932 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca" pod "73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" (UID: "73f18f91-7ec1-4f06-9f23-ca0e3365d4c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:09.332834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.332777 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h" pod "73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" (UID: "73f18f91-7ec1-4f06-9f23-ca0e3365d4c8"). InnerVolumeSpecName "kube-api-access-k2n8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:09.338768 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.338638 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca" pod "73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" (UID: "73f18f91-7ec1-4f06-9f23-ca0e3365d4c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:09.427624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.427582 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2n8h\" (UniqueName: \"kubernetes.io/projected/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-kube-api-access-k2n8h\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.427624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.427619 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.427624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.427628 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.795638 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.795597 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" Oct 29 23:03:09.795878 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.795606 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd" event={"ID":"73f18f91-7ec1-4f06-9f23-ca0e3365d4c8","Type":"ContainerDied","Data":"eb16777d8a0e9c1619378f7f740a1e06f24d2af0f4a09de91aed1d6aaf08af14"} Oct 29 23:03:09.795878 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.795764 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb16777d8a0e9c1619378f7f740a1e06f24d2af0f4a09de91aed1d6aaf08af14" Oct 29 23:03:09.806731 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:09.806682 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" event={"ID":"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f","Type":"ContainerStarted","Data":"e5d3631ef4b5702bd5046a0d16eff9f26d2775ddcf4084314480ee0f171e409e"} Oct 29 23:03:11.225160 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:11.225120 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf566df7-1632-4f99-b188-ded2c6d79545" path="/var/lib/kubelet/pods/bf566df7-1632-4f99-b188-ded2c6d79545/volumes" Oct 29 23:03:11.830291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:11.830251 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" event={"ID":"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f","Type":"ContainerStarted","Data":"cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253"} Oct 29 23:03:11.830516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:11.830358 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:11.870274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:11.870214 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" podStartSLOduration=2.258556088 podStartE2EDuration="3.870199393s" totalImagesPullingTime="1.611643305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:11.8682451 +0000 UTC m=+2858.762320032" watchObservedRunningTime="2026-10-29 23:03:11.870199393 +0000 UTC m=+2858.764274310" Oct 29 23:03:12.029440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:12.029397 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z"] Oct 29 23:03:12.037093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:12.037061 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb98897d-8x25z"] Oct 29 23:03:13.225336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:13.225304 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7aab1d-ff34-4119-a84f-68fb519029d2" path="/var/lib/kubelet/pods/3c7aab1d-ff34-4119-a84f-68fb519029d2/volumes" Oct 29 23:03:16.526282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.526244 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9"] Oct 29 23:03:16.527322 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.527296 5367 cpu_manager.go:414] "containerMap: removing container" podUID="73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" containerName="revoke-keys" Oct 29 23:03:16.527441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.527322 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" containerName="revoke-keys" Oct 29 23:03:16.527561 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.527540 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" containerName="revoke-keys" Oct 29 23:03:16.533500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.533476 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.565504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.565480 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eddb03-openshift-default-dockercfg-tznd2\"" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.639823 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.639940 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvj4j\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640003 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640025 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640158 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640191 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640285 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640347 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.645726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.640576 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.743799 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.743861 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.743949 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.743992 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.744028 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.744073 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.744110 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvj4j\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.744151 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.744169 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.745117 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.745320 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.746239 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.746394 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.748724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.746927 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.749598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.749226 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.770990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.770958 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.775215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.775183 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9"] Oct 29 23:03:16.797795 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.797665 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvj4j\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.823775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.814501 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token\") pod \"e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:16.849546 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:16.848618 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:17.254904 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:17.254863 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff303e1_2b83_4733_b3f2_ed0fb7058a3b.slice/crio-478c09d3c70cb013f020d6b42652bfba70b9a806aaaa0b5cb1da8d5c8800bc22 WatchSource:0}: Error finding container 478c09d3c70cb013f020d6b42652bfba70b9a806aaaa0b5cb1da8d5c8800bc22: Status 404 returned error can't find the container with id 478c09d3c70cb013f020d6b42652bfba70b9a806aaaa0b5cb1da8d5c8800bc22 Oct 29 23:03:17.255337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.255315 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9"] Oct 29 23:03:17.257545 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.257514 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:17.257603 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.257580 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:17.257648 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.257610 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:17.487951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.487911 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:03:17.886381 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.886346 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" event={"ID":"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b","Type":"ContainerStarted","Data":"170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55"} Oct 29 23:03:17.886833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.886394 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" event={"ID":"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b","Type":"ContainerStarted","Data":"478c09d3c70cb013f020d6b42652bfba70b9a806aaaa0b5cb1da8d5c8800bc22"} Oct 29 23:03:17.964195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:17.964141 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podStartSLOduration=1.964126084 podStartE2EDuration="1.964126084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:17.947882695 +0000 UTC m=+2864.841957611" watchObservedRunningTime="2026-10-29 23:03:17.964126084 +0000 UTC m=+2864.858201005" Oct 29 23:03:18.720765 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:18.720715 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd"] Oct 29 23:03:18.732073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:18.732040 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-14b0c3-5v8fd"] Oct 29 23:03:18.854177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:18.854094 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:18.861270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:18.861241 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:18.895311 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:18.895279 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:18.896512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:18.896492 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:03:19.223523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:19.223488 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f18f91-7ec1-4f06-9f23-ca0e3365d4c8" path="/var/lib/kubelet/pods/73f18f91-7ec1-4f06-9f23-ca0e3365d4c8/volumes" Oct 29 23:03:20.917773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:20.917734 5367 generic.go:356] "Generic (PLEG): container finished" podID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerID="5690883cd09be426793c228bb22c289dc660df6e26155fcba22de6a7f146ecd6" exitCode=0 Oct 29 23:03:20.918218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:20.918074 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" event={"ID":"6eae67af-3766-43aa-a36e-b088f5479e2f","Type":"ContainerDied","Data":"5690883cd09be426793c228bb22c289dc660df6e26155fcba22de6a7f146ecd6"} Oct 29 23:03:20.986469 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:20.986441 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 23:03:21.081550 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.081503 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume\") pod \"6eae67af-3766-43aa-a36e-b088f5479e2f\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " Oct 29 23:03:21.081865 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.081566 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h\" (UniqueName: \"kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h\") pod \"6eae67af-3766-43aa-a36e-b088f5479e2f\" (UID: \"6eae67af-3766-43aa-a36e-b088f5479e2f\") " Oct 29 23:03:21.082846 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.082397 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume" pod "6eae67af-3766-43aa-a36e-b088f5479e2f" (UID: "6eae67af-3766-43aa-a36e-b088f5479e2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:21.084283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.084258 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h" pod "6eae67af-3766-43aa-a36e-b088f5479e2f" (UID: "6eae67af-3766-43aa-a36e-b088f5479e2f"). InnerVolumeSpecName "kube-api-access-pd65h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.183959 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.183924 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd65h\" (UniqueName: \"kubernetes.io/projected/6eae67af-3766-43aa-a36e-b088f5479e2f-kube-api-access-pd65h\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.183959 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.183956 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae67af-3766-43aa-a36e-b088f5479e2f-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.928830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.928796 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" Oct 29 23:03:21.928830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.928812 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s" event={"ID":"6eae67af-3766-43aa-a36e-b088f5479e2f","Type":"ContainerDied","Data":"0552cef14841380deaf5981edb29fc2a88f615eba95ac7d1a82297d9bd8af5b5"} Oct 29 23:03:21.929394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.928890 5367 scope.go:122] "RemoveContainer" containerID="5690883cd09be426793c228bb22c289dc660df6e26155fcba22de6a7f146ecd6" Oct 29 23:03:21.984624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.984591 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s"] Oct 29 23:03:21.997166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:21.997127 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d3632-c779784c8-dk28s"] Oct 29 23:03:22.332236 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:22.332193 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:03:22.378223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.378188 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx"] Oct 29 23:03:22.378887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.378871 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerName="payload-processing" Oct 29 23:03:22.378958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.378888 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerName="payload-processing" Oct 29 23:03:22.378997 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.378980 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" containerName="payload-processing" Oct 29 23:03:22.394201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.394174 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.406221 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.406189 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx"] Oct 29 23:03:22.409877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.409850 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:22.410059 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.409865 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:03:22.440548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.440500 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc"] Oct 29 23:03:22.446243 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.446211 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.462807 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.462774 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eddb03\"" Oct 29 23:03:22.467877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.467850 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc"] Oct 29 23:03:22.502005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.501960 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmdc\" (UniqueName: \"kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.502242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.502210 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.502370 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.502351 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.586773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.576450 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eddb03-dockercfg-l5229\"" Oct 29 23:03:22.644718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.640874 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.644718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.640969 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.644718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.641031 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.644718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.641095 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjlk\" (UniqueName: \"kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.644718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.641194 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmdc\" (UniqueName: \"kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.646053 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.645972 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.712341 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.712305 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.714722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.714658 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmdc\" (UniqueName: \"kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc\") pod \"maas-api-revoke-sub-a0d0ac72-qgkbx\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:22.734224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.733115 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6"] Oct 29 23:03:22.739911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.739050 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:22.744266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.744229 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6"] Oct 29 23:03:22.745576 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.745528 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjlk\" (UniqueName: \"kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.745852 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.745828 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqhw\" (UniqueName: \"kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw\") pod \"payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:22.746270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.746243 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:22.746491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.746471 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.747605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.747581 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.765831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.765802 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjlk\" (UniqueName: \"kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:22.842332 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.842253 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:03:22.847417 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.847388 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqhw\" (UniqueName: \"kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw\") pod \"payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:22.847578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.847564 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:22.848285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.848261 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:22.866873 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:22.866827 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqhw\" (UniqueName: \"kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw\") pod \"payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:23.012943 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.012891 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:23.058996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.058962 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:23.064817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.063620 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:23.230222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.230183 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eae67af-3766-43aa-a36e-b088f5479e2f" path="/var/lib/kubelet/pods/6eae67af-3766-43aa-a36e-b088f5479e2f/volumes" Oct 29 23:03:23.409564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.409483 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6"] Oct 29 23:03:23.409564 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:23.409505 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04315dfb_bcff_49e6_8f2d_a64659122722.slice/crio-20f0572c398122777832a300899abd663db6eaebb4c55432a467933135e1d737 WatchSource:0}: Error finding container 20f0572c398122777832a300899abd663db6eaebb4c55432a467933135e1d737: Status 404 returned error can't find the container with id 20f0572c398122777832a300899abd663db6eaebb4c55432a467933135e1d737 Oct 29 23:03:23.412408 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:23.412381 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57dcb70_3557_4506_9459_6aea71fe36e5.slice/crio-c568775fc1db21a9e7bf1b03d0f9749cd9e60f70c12960bca8b809aaa09a60e3 WatchSource:0}: Error finding container c568775fc1db21a9e7bf1b03d0f9749cd9e60f70c12960bca8b809aaa09a60e3: Status 404 returned error can't find the container with id c568775fc1db21a9e7bf1b03d0f9749cd9e60f70c12960bca8b809aaa09a60e3 Oct 29 23:03:23.421243 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.421164 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx"] Oct 29 23:03:23.426480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.426450 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc"] Oct 29 23:03:23.427017 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:23.426994 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ab0a3f_7363_49aa_9ac6_ff1b60437568.slice/crio-539438f44d71c9c2710fdfd9046ad37e283d8bf6dc3c196841ddd21023a0e488 WatchSource:0}: Error finding container 539438f44d71c9c2710fdfd9046ad37e283d8bf6dc3c196841ddd21023a0e488: Status 404 returned error can't find the container with id 539438f44d71c9c2710fdfd9046ad37e283d8bf6dc3c196841ddd21023a0e488 Oct 29 23:03:23.949428 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.949327 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" event={"ID":"a57dcb70-3557-4506-9459-6aea71fe36e5","Type":"ContainerStarted","Data":"28a00438ce31c7dbd75e247ad955d83e2cb5e681963a5c54c5d93e2b372c1b8a"} Oct 29 23:03:23.949428 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.949400 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" event={"ID":"a57dcb70-3557-4506-9459-6aea71fe36e5","Type":"ContainerStarted","Data":"c568775fc1db21a9e7bf1b03d0f9749cd9e60f70c12960bca8b809aaa09a60e3"} Oct 29 23:03:23.951252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.951219 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" event={"ID":"98ab0a3f-7363-49aa-9ac6-ff1b60437568","Type":"ContainerStarted","Data":"6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095"} Oct 29 23:03:23.951645 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.951277 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" event={"ID":"98ab0a3f-7363-49aa-9ac6-ff1b60437568","Type":"ContainerStarted","Data":"539438f44d71c9c2710fdfd9046ad37e283d8bf6dc3c196841ddd21023a0e488"} Oct 29 23:03:23.951645 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.951329 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:23.953215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.953183 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" event={"ID":"04315dfb-bcff-49e6-8f2d-a64659122722","Type":"ContainerStarted","Data":"97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992"} Oct 29 23:03:23.953353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.953238 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" event={"ID":"04315dfb-bcff-49e6-8f2d-a64659122722","Type":"ContainerStarted","Data":"20f0572c398122777832a300899abd663db6eaebb4c55432a467933135e1d737"} Oct 29 23:03:23.953353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.953262 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:23.993535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:23.993470 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" podStartSLOduration=1.993457188 podStartE2EDuration="1.993457188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:23.99144072 +0000 UTC m=+2870.885515642" watchObservedRunningTime="2026-10-29 23:03:23.993457188 +0000 UTC m=+2870.887532108" Oct 29 23:03:24.032331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:24.032270 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" podStartSLOduration=2.03225107 podStartE2EDuration="2.03225107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:24.031393343 +0000 UTC m=+2870.925468265" watchObservedRunningTime="2026-10-29 23:03:24.03225107 +0000 UTC m=+2870.926325996" Oct 29 23:03:24.072047 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:24.071990 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" podStartSLOduration=2.071976623 podStartE2EDuration="2.071976623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:24.070505745 +0000 UTC m=+2870.964580666" watchObservedRunningTime="2026-10-29 23:03:24.071976623 +0000 UTC m=+2870.966051581" Oct 29 23:03:24.964337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:24.964301 5367 generic.go:356] "Generic (PLEG): container finished" podID="a57dcb70-3557-4506-9459-6aea71fe36e5" containerID="28a00438ce31c7dbd75e247ad955d83e2cb5e681963a5c54c5d93e2b372c1b8a" exitCode=0 Oct 29 23:03:24.964527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:24.964387 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" event={"ID":"a57dcb70-3557-4506-9459-6aea71fe36e5","Type":"ContainerDied","Data":"28a00438ce31c7dbd75e247ad955d83e2cb5e681963a5c54c5d93e2b372c1b8a"} Oct 29 23:03:26.080288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.080228 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s"] Oct 29 23:03:26.080830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.080588 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" containerID="cri-o://555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc" gracePeriod=30 Oct 29 23:03:26.301851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.294127 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:26.314181 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.314138 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca\") pod \"a57dcb70-3557-4506-9459-6aea71fe36e5\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " Oct 29 23:03:26.315934 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.315904 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca\") pod \"a57dcb70-3557-4506-9459-6aea71fe36e5\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " Oct 29 23:03:26.316087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.316070 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc\" (UniqueName: \"kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc\") pod \"a57dcb70-3557-4506-9459-6aea71fe36e5\" (UID: \"a57dcb70-3557-4506-9459-6aea71fe36e5\") " Oct 29 23:03:26.317200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.317175 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca" pod "a57dcb70-3557-4506-9459-6aea71fe36e5" (UID: "a57dcb70-3557-4506-9459-6aea71fe36e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:26.333083 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.333022 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc" pod "a57dcb70-3557-4506-9459-6aea71fe36e5" (UID: "a57dcb70-3557-4506-9459-6aea71fe36e5"). InnerVolumeSpecName "kube-api-access-9dmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:26.335215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.335185 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca" pod "a57dcb70-3557-4506-9459-6aea71fe36e5" (UID: "a57dcb70-3557-4506-9459-6aea71fe36e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:26.418191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.418151 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:26.418191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.418186 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmdc\" (UniqueName: \"kubernetes.io/projected/a57dcb70-3557-4506-9459-6aea71fe36e5-kube-api-access-9dmdc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:26.418191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.418197 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57dcb70-3557-4506-9459-6aea71fe36e5-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:26.592177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.592131 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5"] Oct 29 23:03:26.592877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.592855 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a57dcb70-3557-4506-9459-6aea71fe36e5" containerName="revoke-keys" Oct 29 23:03:26.592877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.592872 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57dcb70-3557-4506-9459-6aea71fe36e5" containerName="revoke-keys" Oct 29 23:03:26.593061 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.592985 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57dcb70-3557-4506-9459-6aea71fe36e5" containerName="revoke-keys" Oct 29 23:03:26.596684 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.596665 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.621046 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.621008 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5"] Oct 29 23:03:26.629212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.629175 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.629473 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.629438 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.629598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.629575 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n252\" (UniqueName: \"kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.730886 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.730846 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.731073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.730901 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.731073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.731005 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n252\" (UniqueName: \"kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.731601 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.731577 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.733307 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.733282 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.751667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.751638 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n252\" (UniqueName: \"kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:26.911308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:26.911212 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:27.049901 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:27.049872 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" event={"ID":"a57dcb70-3557-4506-9459-6aea71fe36e5","Type":"ContainerDied","Data":"c568775fc1db21a9e7bf1b03d0f9749cd9e60f70c12960bca8b809aaa09a60e3"} Oct 29 23:03:27.050111 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:27.050100 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c568775fc1db21a9e7bf1b03d0f9749cd9e60f70c12960bca8b809aaa09a60e3" Oct 29 23:03:27.050240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:27.050231 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx" Oct 29 23:03:27.230451 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:27.230413 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05954cd5_636d_405d_beda_25c3996239a1.slice/crio-9e0b54623dcb11dc74cd043eea81b63b779b3f276cc242d7204638a3203bf322 WatchSource:0}: Error finding container 9e0b54623dcb11dc74cd043eea81b63b779b3f276cc242d7204638a3203bf322: Status 404 returned error can't find the container with id 9e0b54623dcb11dc74cd043eea81b63b779b3f276cc242d7204638a3203bf322 Oct 29 23:03:27.231256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:27.231233 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5"] Oct 29 23:03:28.062900 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:28.062857 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" event={"ID":"05954cd5-636d-405d-beda-25c3996239a1","Type":"ContainerStarted","Data":"9a5e8a748e4094f29ab70743c5a1f1b04352af5c0c2d4f446e00804f0068ad6c"} Oct 29 23:03:28.063083 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:28.062915 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" event={"ID":"05954cd5-636d-405d-beda-25c3996239a1","Type":"ContainerStarted","Data":"9e0b54623dcb11dc74cd043eea81b63b779b3f276cc242d7204638a3203bf322"} Oct 29 23:03:28.109133 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:28.109078 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" podStartSLOduration=2.109062302 podStartE2EDuration="2.109062302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:28.104873067 +0000 UTC m=+2874.998947989" watchObservedRunningTime="2026-10-29 23:03:28.109062302 +0000 UTC m=+2875.003137222" Oct 29 23:03:29.018092 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.018056 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn"] Oct 29 23:03:29.022392 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.022369 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.038855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.038830 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c74e55-openshift-default-dockercfg-zchv6\"" Oct 29 23:03:29.055552 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.055512 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn"] Oct 29 23:03:29.071378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071340 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071388 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071463 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071528 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071666 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071715 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9lm\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071835 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071745 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071864 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.071927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.071895 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.076015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.075981 5367 generic.go:356] "Generic (PLEG): container finished" podID="05954cd5-636d-405d-beda-25c3996239a1" containerID="9a5e8a748e4094f29ab70743c5a1f1b04352af5c0c2d4f446e00804f0068ad6c" exitCode=0 Oct 29 23:03:29.076162 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.076059 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" event={"ID":"05954cd5-636d-405d-beda-25c3996239a1","Type":"ContainerDied","Data":"9a5e8a748e4094f29ab70743c5a1f1b04352af5c0c2d4f446e00804f0068ad6c"} Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199240 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199297 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199382 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199456 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199614 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199648 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9lm\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199673 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199820 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.201722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.199850 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.257724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.252771 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.257724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.253077 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.257724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.253343 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.257724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.254080 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.290773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.287543 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.326722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.324032 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9lm\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.326722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.324690 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.329780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.328435 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.339066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.335749 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo\") pod \"e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.340732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.340676 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:29.681121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.681089 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn"] Oct 29 23:03:29.682950 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:29.682925 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeee6908_14c0_4bde_ad79_ee142dd07a29.slice/crio-3859de540e904ce892a83294fb2eea2f099dd8e584f30dd9f76f2fcdb7db6a50 WatchSource:0}: Error finding container 3859de540e904ce892a83294fb2eea2f099dd8e584f30dd9f76f2fcdb7db6a50: Status 404 returned error can't find the container with id 3859de540e904ce892a83294fb2eea2f099dd8e584f30dd9f76f2fcdb7db6a50 Oct 29 23:03:29.686605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.686567 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:29.686725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.686640 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:29.686725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:29.686669 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:30.087298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.087252 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" event={"ID":"beee6908-14c0-4bde-ad79-ee142dd07a29","Type":"ContainerStarted","Data":"b85aa3e689949d24345d952346ac74baf4e193e6a3b87eae30e8790363488d64"} Oct 29 23:03:30.087784 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.087319 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" event={"ID":"beee6908-14c0-4bde-ad79-ee142dd07a29","Type":"ContainerStarted","Data":"3859de540e904ce892a83294fb2eea2f099dd8e584f30dd9f76f2fcdb7db6a50"} Oct 29 23:03:30.134551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.134490 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podStartSLOduration=2.134472246 podStartE2EDuration="2.134472246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:30.131268559 +0000 UTC m=+2877.025343491" watchObservedRunningTime="2026-10-29 23:03:30.134472246 +0000 UTC m=+2877.028547166" Oct 29 23:03:30.149937 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.149902 5367 patch_prober.go:29] interesting pod/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:03:30.150125 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.149959 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:03:30.232437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.232408 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:30.346140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.340744 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca\") pod \"05954cd5-636d-405d-beda-25c3996239a1\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " Oct 29 23:03:30.346140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.340922 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252\" (UniqueName: \"kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252\") pod \"05954cd5-636d-405d-beda-25c3996239a1\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " Oct 29 23:03:30.346140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.341122 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca\") pod \"05954cd5-636d-405d-beda-25c3996239a1\" (UID: \"05954cd5-636d-405d-beda-25c3996239a1\") " Oct 29 23:03:30.346140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.345024 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca" pod "05954cd5-636d-405d-beda-25c3996239a1" (UID: "05954cd5-636d-405d-beda-25c3996239a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:30.346516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.346352 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:30.382775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.380248 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:30.383217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.383182 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca" pod "05954cd5-636d-405d-beda-25c3996239a1" (UID: "05954cd5-636d-405d-beda-25c3996239a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:30.385734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.385384 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252" pod "05954cd5-636d-405d-beda-25c3996239a1" (UID: "05954cd5-636d-405d-beda-25c3996239a1"). InnerVolumeSpecName "kube-api-access-7n252". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:30.443338 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.443300 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05954cd5-636d-405d-beda-25c3996239a1-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:30.443338 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.443335 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n252\" (UniqueName: \"kubernetes.io/projected/05954cd5-636d-405d-beda-25c3996239a1-kube-api-access-7n252\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:30.443566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:30.443349 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05954cd5-636d-405d-beda-25c3996239a1-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.097686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.097554 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" Oct 29 23:03:31.097686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.097602 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5" event={"ID":"05954cd5-636d-405d-beda-25c3996239a1","Type":"ContainerDied","Data":"9e0b54623dcb11dc74cd043eea81b63b779b3f276cc242d7204638a3203bf322"} Oct 29 23:03:31.097686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.097655 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e0b54623dcb11dc74cd043eea81b63b779b3f276cc242d7204638a3203bf322" Oct 29 23:03:31.098125 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.098012 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:31.099265 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.099233 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:03:31.404066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.404041 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 23:03:31.560732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560673 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.560906 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560773 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.560906 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560803 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.560906 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560828 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.561110 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560913 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.561110 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560949 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.561110 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.560963 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:31.561110 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561095 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:31.561337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561151 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.561337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561196 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.561337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561206 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:31.561337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561241 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token\") pod \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\" (UID: \"43abfea5-a446-4ccf-be1b-b0dbcc19a091\") " Oct 29 23:03:31.561550 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561339 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:31.561550 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561422 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:31.561970 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561943 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.562097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561973 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.562097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.561987 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.562097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.562005 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.562097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.562018 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.563593 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.563563 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:31.563694 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.563613 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "kube-api-access-248q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:31.563694 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.563640 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:31.565541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.565522 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy" pod "43abfea5-a446-4ccf-be1b-b0dbcc19a091" (UID: "43abfea5-a446-4ccf-be1b-b0dbcc19a091"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:31.663578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.663500 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.663578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.663526 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248q4\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-kube-api-access-248q4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.663578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.663540 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:31.663578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:31.663550 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43abfea5-a446-4ccf-be1b-b0dbcc19a091-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:32.110436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.110383 5367 generic.go:356] "Generic (PLEG): container finished" podID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerID="555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc" exitCode=0 Oct 29 23:03:32.110958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.110459 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" Oct 29 23:03:32.110958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.110472 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" event={"ID":"43abfea5-a446-4ccf-be1b-b0dbcc19a091","Type":"ContainerDied","Data":"555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc"} Oct 29 23:03:32.110958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.110556 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s" event={"ID":"43abfea5-a446-4ccf-be1b-b0dbcc19a091","Type":"ContainerDied","Data":"6f59ec2a6c65b746a29a18b200b907578b0d71d83d880cd689861e99310194bf"} Oct 29 23:03:32.110958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.110592 5367 scope.go:122] "RemoveContainer" containerID="555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc" Oct 29 23:03:32.124504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.124483 5367 scope.go:122] "RemoveContainer" containerID="d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543" Oct 29 23:03:32.135266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.135241 5367 scope.go:122] "RemoveContainer" containerID="555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc" Oct 29 23:03:32.135595 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:32.135575 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc\": container with ID starting with 555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc not found: ID does not exist" containerID="555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc" Oct 29 23:03:32.135669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.135611 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc"} err="failed to get container status \"555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc\": rpc error: code = NotFound desc = could not find container \"555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc\": container with ID starting with 555e1cd1d1c93541d65afb8157f624800f7309c692100fd7162bfd65a3a986bc not found: ID does not exist" Oct 29 23:03:32.135669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.135635 5367 scope.go:122] "RemoveContainer" containerID="d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543" Oct 29 23:03:32.135922 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:32.135900 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543\": container with ID starting with d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543 not found: ID does not exist" containerID="d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543" Oct 29 23:03:32.135975 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.135931 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543"} err="failed to get container status \"d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543\": rpc error: code = NotFound desc = could not find container \"d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543\": container with ID starting with d652daa866f4b030917b4248f32c5cbb361c4f7bc219be01e8d64ab506625543 not found: ID does not exist" Oct 29 23:03:32.169596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.169557 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s"] Oct 29 23:03:32.179916 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:32.179874 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-14b0c3-openshift-default-846bddfb8f-7cr4s"] Oct 29 23:03:33.224960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:33.224928 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" path="/var/lib/kubelet/pods/43abfea5-a446-4ccf-be1b-b0dbcc19a091/volumes" Oct 29 23:03:34.968021 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:34.967993 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:03:34.968535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:34.968075 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:03:34.969329 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:34.969311 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt"] Oct 29 23:03:34.973230 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:34.973208 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d3632-nvxvt"] Oct 29 23:03:35.124224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.124196 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld"] Oct 29 23:03:35.124960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.124939 5367 cpu_manager.go:414] "containerMap: removing container" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" Oct 29 23:03:35.124960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.124960 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" Oct 29 23:03:35.125153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.124976 5367 cpu_manager.go:414] "containerMap: removing container" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" Oct 29 23:03:35.125153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.124983 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" Oct 29 23:03:35.125153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.125025 5367 cpu_manager.go:414] "containerMap: removing container" podUID="05954cd5-636d-405d-beda-25c3996239a1" containerName="revoke-keys" Oct 29 23:03:35.125153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.125033 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05954cd5-636d-405d-beda-25c3996239a1" containerName="revoke-keys" Oct 29 23:03:35.125346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.125211 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="05954cd5-636d-405d-beda-25c3996239a1" containerName="revoke-keys" Oct 29 23:03:35.125346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.125233 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" Oct 29 23:03:35.125346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.125244 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="43abfea5-a446-4ccf-be1b-b0dbcc19a091" containerName="istio-proxy" Oct 29 23:03:35.130063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.130041 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.145511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.145484 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c74e55-dockercfg-q82fw\"" Oct 29 23:03:35.146314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.146283 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c74e55\"" Oct 29 23:03:35.148537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.148515 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld"] Oct 29 23:03:35.242773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.201580 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gnr\" (UniqueName: \"kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.263862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.263829 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.380728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.365545 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gnr\" (UniqueName: \"kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.380728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.365659 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.380728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.367520 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.380728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.370141 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31f7468-f5fe-4dbe-bf70-8956854b4c3e" path="/var/lib/kubelet/pods/a31f7468-f5fe-4dbe-bf70-8956854b4c3e/volumes" Oct 29 23:03:35.400777 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.398222 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gnr\" (UniqueName: \"kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.444961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.444926 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:35.756623 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:35.756548 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7363cfa8_d323_4ebd_a3c6_44497d2ff1ee.slice/crio-e206d61178d595f98d7dbc854f7822ebcb9b51284b8c883be5caf0ec7ebce068 WatchSource:0}: Error finding container e206d61178d595f98d7dbc854f7822ebcb9b51284b8c883be5caf0ec7ebce068: Status 404 returned error can't find the container with id e206d61178d595f98d7dbc854f7822ebcb9b51284b8c883be5caf0ec7ebce068 Oct 29 23:03:35.756808 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:35.756676 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld"] Oct 29 23:03:36.174775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:36.174737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" event={"ID":"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee","Type":"ContainerStarted","Data":"696f724e6b83de2b23ba4b8f4343fbbe2f88983fba72bff8d460200abd5bfc6d"} Oct 29 23:03:36.175254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:36.174795 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" event={"ID":"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee","Type":"ContainerStarted","Data":"e206d61178d595f98d7dbc854f7822ebcb9b51284b8c883be5caf0ec7ebce068"} Oct 29 23:03:36.175254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:36.174819 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:36.226723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:36.226360 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" podStartSLOduration=1.226338712 podStartE2EDuration="1.226338712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:36.21803218 +0000 UTC m=+2883.112107105" watchObservedRunningTime="2026-10-29 23:03:36.226338712 +0000 UTC m=+2883.120413635" Oct 29 23:03:39.130152 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:39.130116 5367 scope.go:122] "RemoveContainer" containerID="173737229e1303e675b5608027266272843856a3f491aa7165ff98913b247518" Oct 29 23:03:41.876680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:41.876641 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs"] Oct 29 23:03:41.877120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:41.876933 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" containerID="cri-o://359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0" gracePeriod=30 Oct 29 23:03:47.151621 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.151592 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 23:03:47.188392 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.188327 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:03:47.237035 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.236998 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237084 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237170 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237233 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237225 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237287 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237368 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237413 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237458 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.237764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.237498 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert\") pod \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\" (UID: \"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f\") " Oct 29 23:03:47.238291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.238262 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.239611 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.239587 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.240234 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.239802 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.240369 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.239815 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:47.240468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.240193 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.241538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.241517 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "kube-api-access-llnk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:47.243009 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.242988 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.243120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.242992 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:47.244456 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.244432 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token" pod "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" (UID: "c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:47.272348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.272314 5367 generic.go:356] "Generic (PLEG): container finished" podID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerID="359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0" exitCode=0 Oct 29 23:03:47.272544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.272374 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" Oct 29 23:03:47.272544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.272391 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" event={"ID":"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f","Type":"ContainerDied","Data":"359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0"} Oct 29 23:03:47.272544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.272456 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs" event={"ID":"c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f","Type":"ContainerDied","Data":"0216b87face5d4550610d3245f64fc9d38a6fb29b14db46c31980f2592daa3c0"} Oct 29 23:03:47.272544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.272478 5367 scope.go:122] "RemoveContainer" containerID="359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0" Oct 29 23:03:47.286238 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.286220 5367 scope.go:122] "RemoveContainer" containerID="359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0" Oct 29 23:03:47.286557 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:47.286539 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0\": container with ID starting with 359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0 not found: ID does not exist" containerID="359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0" Oct 29 23:03:47.286643 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.286567 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0"} err="failed to get container status \"359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0\": rpc error: code = NotFound desc = could not find container \"359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0\": container with ID starting with 359da1443d8b86f8d29f19f961e30ae6eee923a518759b54908a2f1ea70eb9f0 not found: ID does not exist" Oct 29 23:03:47.328548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.328521 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs"] Oct 29 23:03:47.335750 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.335719 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-zpmjs"] Oct 29 23:03:47.339486 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339460 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339491 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339505 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339519 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339531 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339546 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llnk6\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-kube-api-access-llnk6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339561 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339574 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:47.339587 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:49.222628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:49.222594 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" path="/var/lib/kubelet/pods/c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f/volumes" Oct 29 23:03:51.164806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.164770 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn"] Oct 29 23:03:51.165458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.165444 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" Oct 29 23:03:51.165512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.165460 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" Oct 29 23:03:51.165577 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.165568 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cd45f2-b51b-4edb-a5eb-3c66a9a5193f" containerName="istio-proxy" Oct 29 23:03:51.170914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.170895 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.191310 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.191282 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn"] Oct 29 23:03:51.202050 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.202017 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eddb03-serving-cert\"" Oct 29 23:03:51.303972 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.303926 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqj8\" (UniqueName: \"kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.304158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.304050 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.304220 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.304169 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.304319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.304233 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.405328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.405284 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqj8\" (UniqueName: \"kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.405541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.405398 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.405623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.405544 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.405623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.405585 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.406238 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.406204 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.407900 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.407874 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.408008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.407970 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.426514 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.426451 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqj8\" (UniqueName: \"kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8\") pod \"maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.487545 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.487509 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:51.798053 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:03:51.797997 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a50fe23_eb39_45a2_ba51_8db99c581762.slice/crio-b4204d23e95ded17541e8ed622bbb4ea21ae4eac27dd8e89640e6f9de730bc3c WatchSource:0}: Error finding container b4204d23e95ded17541e8ed622bbb4ea21ae4eac27dd8e89640e6f9de730bc3c: Status 404 returned error can't find the container with id b4204d23e95ded17541e8ed622bbb4ea21ae4eac27dd8e89640e6f9de730bc3c Oct 29 23:03:51.798445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:51.798420 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn"] Oct 29 23:03:52.316189 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.316145 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" event={"ID":"8a50fe23-eb39-45a2-ba51-8db99c581762","Type":"ContainerStarted","Data":"b4204d23e95ded17541e8ed622bbb4ea21ae4eac27dd8e89640e6f9de730bc3c"} Oct 29 23:03:52.332471 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:52.332439 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:03:52.456436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.456401 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78"] Oct 29 23:03:52.456815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.456642 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" podUID="95c75367-0bea-44db-a48f-424299e6e3e5" containerName="maas-api" containerID="cri-o://db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0" gracePeriod=30 Oct 29 23:03:52.768805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.768780 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:03:52.850099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.850026 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca\") pod \"95c75367-0bea-44db-a48f-424299e6e3e5\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " Oct 29 23:03:52.850246 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.850160 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs\") pod \"95c75367-0bea-44db-a48f-424299e6e3e5\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " Oct 29 23:03:52.850317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.850256 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls\") pod \"95c75367-0bea-44db-a48f-424299e6e3e5\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " Oct 29 23:03:52.850444 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.850421 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca" pod "95c75367-0bea-44db-a48f-424299e6e3e5" (UID: "95c75367-0bea-44db-a48f-424299e6e3e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:52.850529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.850454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t\" (UniqueName: \"kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t\") pod \"95c75367-0bea-44db-a48f-424299e6e3e5\" (UID: \"95c75367-0bea-44db-a48f-424299e6e3e5\") " Oct 29 23:03:52.851127 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.851100 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c75367-0bea-44db-a48f-424299e6e3e5-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.852687 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.852667 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs" pod "95c75367-0bea-44db-a48f-424299e6e3e5" (UID: "95c75367-0bea-44db-a48f-424299e6e3e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:52.853135 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.853113 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls" pod "95c75367-0bea-44db-a48f-424299e6e3e5" (UID: "95c75367-0bea-44db-a48f-424299e6e3e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:52.853484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.853465 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t" pod "95c75367-0bea-44db-a48f-424299e6e3e5" (UID: "95c75367-0bea-44db-a48f-424299e6e3e5"). InnerVolumeSpecName "kube-api-access-v5k4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:52.952372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.952339 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.952372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.952369 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c75367-0bea-44db-a48f-424299e6e3e5-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.952577 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:52.952381 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5k4t\" (UniqueName: \"kubernetes.io/projected/95c75367-0bea-44db-a48f-424299e6e3e5-kube-api-access-v5k4t\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.099880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.099847 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5"] Oct 29 23:03:53.103756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.103654 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d286f-ftdw5"] Oct 29 23:03:53.227343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.227306 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05954cd5-636d-405d-beda-25c3996239a1" path="/var/lib/kubelet/pods/05954cd5-636d-405d-beda-25c3996239a1/volumes" Oct 29 23:03:53.367777 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.367137 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" event={"ID":"8a50fe23-eb39-45a2-ba51-8db99c581762","Type":"ContainerStarted","Data":"a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41"} Oct 29 23:03:53.368254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.368122 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:03:53.398522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.393968 5367 generic.go:356] "Generic (PLEG): container finished" podID="95c75367-0bea-44db-a48f-424299e6e3e5" containerID="db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0" exitCode=0 Oct 29 23:03:53.398522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.394110 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" Oct 29 23:03:53.398522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.394222 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" event={"ID":"95c75367-0bea-44db-a48f-424299e6e3e5","Type":"ContainerDied","Data":"db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0"} Oct 29 23:03:53.398522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.394286 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78" event={"ID":"95c75367-0bea-44db-a48f-424299e6e3e5","Type":"ContainerDied","Data":"c3d0114b07c2cde05ffd66fbfdfae833572b2c91f67445c565b377036d1ad93f"} Oct 29 23:03:53.398522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.394319 5367 scope.go:122] "RemoveContainer" containerID="db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0" Oct 29 23:03:53.412854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.412754 5367 scope.go:122] "RemoveContainer" containerID="db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0" Oct 29 23:03:53.413246 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:03:53.413219 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0\": container with ID starting with db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0 not found: ID does not exist" containerID="db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0" Oct 29 23:03:53.413326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.413259 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0"} err="failed to get container status \"db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0\": rpc error: code = NotFound desc = could not find container \"db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0\": container with ID starting with db8e1a38530d06aa15e7b8b64312356f838083954df6f7686d1c03e3d20368f0 not found: ID does not exist" Oct 29 23:03:53.423158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.423110 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" podStartSLOduration=1.805748991 podStartE2EDuration="2.423094567s" totalImagesPullingTime="617.345576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:53.417131786 +0000 UTC m=+2900.311206709" watchObservedRunningTime="2026-10-29 23:03:53.423094567 +0000 UTC m=+2900.317169487" Oct 29 23:03:53.459432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.459392 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78"] Oct 29 23:03:53.463480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.463452 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d286f-5c8b9c66c8-ncl78"] Oct 29 23:03:53.673471 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.673384 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7"] Oct 29 23:03:53.677641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:53.677609 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dabb8-dmnz7"] Oct 29 23:03:55.223029 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:55.222994 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa13ab7-8f20-4ee1-9b80-e74a428adaaf" path="/var/lib/kubelet/pods/2fa13ab7-8f20-4ee1-9b80-e74a428adaaf/volumes" Oct 29 23:03:55.223449 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:03:55.223434 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c75367-0bea-44db-a48f-424299e6e3e5" path="/var/lib/kubelet/pods/95c75367-0bea-44db-a48f-424299e6e3e5/volumes" Oct 29 23:04:00.456723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.454040 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:04:00.789455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.789374 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq"] Oct 29 23:04:00.790040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.790026 5367 cpu_manager.go:414] "containerMap: removing container" podUID="95c75367-0bea-44db-a48f-424299e6e3e5" containerName="maas-api" Oct 29 23:04:00.790091 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.790041 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c75367-0bea-44db-a48f-424299e6e3e5" containerName="maas-api" Oct 29 23:04:00.790161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.790152 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c75367-0bea-44db-a48f-424299e6e3e5" containerName="maas-api" Oct 29 23:04:00.795212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.795194 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:00.813602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.813576 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:00.813794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.813617 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:04:00.815682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.815660 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq"] Oct 29 23:04:00.913270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.913237 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:00.913455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.913436 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:00.913503 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:00.913477 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djk2\" (UniqueName: \"kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.015441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.015409 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.015657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.015462 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djk2\" (UniqueName: \"kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.015657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.015511 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.016352 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.016328 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.018089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.018059 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.037565 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.037538 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djk2\" (UniqueName: \"kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.109052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.109021 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:01.416223 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:04:01.416185 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2054ad8a_4dba_4bc5_bff1_adad417d96e4.slice/crio-d750a1f96324bbf221bd574c9bb9e616600f1612cb1f06920357b490af236e91 WatchSource:0}: Error finding container d750a1f96324bbf221bd574c9bb9e616600f1612cb1f06920357b490af236e91: Status 404 returned error can't find the container with id d750a1f96324bbf221bd574c9bb9e616600f1612cb1f06920357b490af236e91 Oct 29 23:04:01.416652 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.416629 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq"] Oct 29 23:04:01.479435 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:01.479399 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" event={"ID":"2054ad8a-4dba-4bc5-bff1-adad417d96e4","Type":"ContainerStarted","Data":"d750a1f96324bbf221bd574c9bb9e616600f1612cb1f06920357b490af236e91"} Oct 29 23:04:02.490101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:02.490007 5367 generic.go:356] "Generic (PLEG): container finished" podID="2054ad8a-4dba-4bc5-bff1-adad417d96e4" containerID="24c9aea470c56e231e1b4735ada66a67cc96fb35833662e2083b6ca53674231b" exitCode=0 Oct 29 23:04:02.490101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:02.490084 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" event={"ID":"2054ad8a-4dba-4bc5-bff1-adad417d96e4","Type":"ContainerDied","Data":"24c9aea470c56e231e1b4735ada66a67cc96fb35833662e2083b6ca53674231b"} Oct 29 23:04:03.663164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.663135 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:03.794271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.794172 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2\" (UniqueName: \"kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2\") pod \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " Oct 29 23:04:03.802111 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.802065 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca\") pod \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " Oct 29 23:04:03.823010 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.820962 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca" pod "2054ad8a-4dba-4bc5-bff1-adad417d96e4" (UID: "2054ad8a-4dba-4bc5-bff1-adad417d96e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:03.823850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.823376 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca\") pod \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\" (UID: \"2054ad8a-4dba-4bc5-bff1-adad417d96e4\") " Oct 29 23:04:03.838733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.837638 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:03.839079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.839038 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca" pod "2054ad8a-4dba-4bc5-bff1-adad417d96e4" (UID: "2054ad8a-4dba-4bc5-bff1-adad417d96e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:03.839196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.839050 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2" pod "2054ad8a-4dba-4bc5-bff1-adad417d96e4" (UID: "2054ad8a-4dba-4bc5-bff1-adad417d96e4"). InnerVolumeSpecName "kube-api-access-8djk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:03.938650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.938615 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2054ad8a-4dba-4bc5-bff1-adad417d96e4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:03.938650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:03.938646 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djk2\" (UniqueName: \"kubernetes.io/projected/2054ad8a-4dba-4bc5-bff1-adad417d96e4-kube-api-access-8djk2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:04.517423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:04.517388 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" Oct 29 23:04:04.517620 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:04.517434 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq" event={"ID":"2054ad8a-4dba-4bc5-bff1-adad417d96e4","Type":"ContainerDied","Data":"d750a1f96324bbf221bd574c9bb9e616600f1612cb1f06920357b490af236e91"} Oct 29 23:04:04.517620 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:04.517494 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d750a1f96324bbf221bd574c9bb9e616600f1612cb1f06920357b490af236e91" Oct 29 23:04:11.607136 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:11.607101 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv"] Oct 29 23:04:11.607690 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:11.607308 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" containerID="cri-o://45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de" gracePeriod=30 Oct 29 23:04:17.023340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.023309 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 23:04:17.115271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115238 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115431 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115298 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115431 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115341 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115431 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115411 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115463 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115480 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115499 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115539 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.115731 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.115680 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket\") pod \"730090a7-005a-414c-9764-1313ac482dfc\" (UID: \"730090a7-005a-414c-9764-1313ac482dfc\") " Oct 29 23:04:17.116334 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.116306 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.116447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.116386 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.116527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.116435 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.116527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.116497 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.116757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.116735 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.119950 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.119914 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.119950 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.119932 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.120258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.120187 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.120258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.120230 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2" pod "730090a7-005a-414c-9764-1313ac482dfc" (UID: "730090a7-005a-414c-9764-1313ac482dfc"). InnerVolumeSpecName "kube-api-access-9lhk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.219021 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.218932 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219231 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219369 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219357 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219468 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219579 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219566 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730090a7-005a-414c-9764-1313ac482dfc-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219654 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219768 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730090a7-005a-414c-9764-1313ac482dfc-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219862 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730090a7-005a-414c-9764-1313ac482dfc-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.219980 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.219966 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhk2\" (UniqueName: \"kubernetes.io/projected/730090a7-005a-414c-9764-1313ac482dfc-kube-api-access-9lhk2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.658255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.658218 5367 generic.go:356] "Generic (PLEG): container finished" podID="730090a7-005a-414c-9764-1313ac482dfc" containerID="45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de" exitCode=0 Oct 29 23:04:17.658777 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.658750 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" event={"ID":"730090a7-005a-414c-9764-1313ac482dfc","Type":"ContainerDied","Data":"45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de"} Oct 29 23:04:17.658984 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.658964 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" event={"ID":"730090a7-005a-414c-9764-1313ac482dfc","Type":"ContainerDied","Data":"aabfad1fd48b0cbbd4d33b0064155b81b88166f56e24bebe96a681511441533e"} Oct 29 23:04:17.659130 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.659116 5367 scope.go:122] "RemoveContainer" containerID="45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de" Oct 29 23:04:17.659430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.659410 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv" Oct 29 23:04:17.714802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.714779 5367 scope.go:122] "RemoveContainer" containerID="45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de" Oct 29 23:04:17.715105 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:04:17.715087 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de\": container with ID starting with 45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de not found: ID does not exist" containerID="45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de" Oct 29 23:04:17.715205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.715189 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de"} err="failed to get container status \"45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de\": rpc error: code = NotFound desc = could not find container \"45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de\": container with ID starting with 45ccf9abd0c578ae82d2ae4273323f87ff5caaf375eaf3330b77a0c942a261de not found: ID does not exist" Oct 29 23:04:17.743939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.743908 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv"] Oct 29 23:04:17.803523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:17.803420 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-wsnwv"] Oct 29 23:04:19.222153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:19.222109 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730090a7-005a-414c-9764-1313ac482dfc" path="/var/lib/kubelet/pods/730090a7-005a-414c-9764-1313ac482dfc/volumes" Oct 29 23:04:20.762730 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:20.762676 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq"] Oct 29 23:04:20.776611 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:20.776573 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-437981-fl8wq"] Oct 29 23:04:21.229081 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:21.229045 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2054ad8a-4dba-4bc5-bff1-adad417d96e4" path="/var/lib/kubelet/pods/2054ad8a-4dba-4bc5-bff1-adad417d96e4/volumes" Oct 29 23:04:22.335978 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:04:22.335939 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:04:24.784774 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:24.784745 5367 generic.go:356] "Generic (PLEG): container finished" podID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerID="b85aa3e689949d24345d952346ac74baf4e193e6a3b87eae30e8790363488d64" exitCode=0 Oct 29 23:04:24.785257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:24.784824 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" event={"ID":"beee6908-14c0-4bde-ad79-ee142dd07a29","Type":"ContainerDied","Data":"b85aa3e689949d24345d952346ac74baf4e193e6a3b87eae30e8790363488d64"} Oct 29 23:04:24.785383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:24.785364 5367 scope.go:122] "RemoveContainer" containerID="b85aa3e689949d24345d952346ac74baf4e193e6a3b87eae30e8790363488d64" Oct 29 23:04:24.787013 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:24.786985 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:24.787099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:24.787052 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:24.787099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:24.787089 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:25.795171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.795136 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" event={"ID":"beee6908-14c0-4bde-ad79-ee142dd07a29","Type":"ContainerStarted","Data":"90b201e3b953fcf6b08a5c379a8b668a52100f0a67f1499592ed14841c76910a"} Oct 29 23:04:25.797125 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.797100 5367 generic.go:356] "Generic (PLEG): container finished" podID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerID="170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55" exitCode=0 Oct 29 23:04:25.797245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.797178 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" event={"ID":"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b","Type":"ContainerDied","Data":"170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55"} Oct 29 23:04:25.797560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.797547 5367 scope.go:122] "RemoveContainer" containerID="170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55" Oct 29 23:04:25.799289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.799261 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:25.799376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.799331 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:25.799376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:25.799365 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:26.341608 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.341576 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:04:26.341836 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.341816 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 23:04:26.341907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.341876 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 23:04:26.814771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.814736 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" event={"ID":"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b","Type":"ContainerStarted","Data":"8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf"} Oct 29 23:04:26.850598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.850568 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:04:26.850794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.850650 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 29 23:04:26.850794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.850680 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 29 23:04:27.009793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:26.991693 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6"] Oct 29 23:04:27.014955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.014927 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc"] Oct 29 23:04:27.341336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.341297 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 23:04:27.341531 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.341352 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 23:04:27.729994 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.729904 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv"] Oct 29 23:04:27.730178 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.730125 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" containerID="cri-o://4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa" gracePeriod=30 Oct 29 23:04:27.870135 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.866850 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 29 23:04:27.870135 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:27.866912 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 29 23:04:28.341344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.341293 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 23:04:28.341569 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.341358 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 23:04:28.377750 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.377356 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24"] Oct 29 23:04:28.381727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.378447 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2054ad8a-4dba-4bc5-bff1-adad417d96e4" containerName="revoke-keys" Oct 29 23:04:28.381727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.378475 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2054ad8a-4dba-4bc5-bff1-adad417d96e4" containerName="revoke-keys" Oct 29 23:04:28.381727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.378509 5367 cpu_manager.go:414] "containerMap: removing container" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" Oct 29 23:04:28.381727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.378518 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" Oct 29 23:04:28.381727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.378685 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2054ad8a-4dba-4bc5-bff1-adad417d96e4" containerName="revoke-keys" Oct 29 23:04:28.397551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.394754 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="730090a7-005a-414c-9764-1313ac482dfc" containerName="istio-proxy" Oct 29 23:04:28.407015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.406986 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.410313 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.410279 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24"] Oct 29 23:04:28.424975 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.424953 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:04:28.425170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.425145 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:28.449801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.449772 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.449983 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.449965 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzcd\" (UniqueName: \"kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.450054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.450035 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.551099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.551056 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.551311 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.551156 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.551395 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.551347 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzcd\" (UniqueName: \"kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.551880 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.551853 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.553609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.553586 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.595765 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.594367 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzcd\" (UniqueName: \"kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.727115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.727080 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:28.850668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.850483 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 29 23:04:28.850668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:28.850557 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 29 23:04:29.107211 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.107138 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24"] Oct 29 23:04:29.107914 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:04:29.107883 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a45c32_e48d_4d51_824a_e9aa89e3f701.slice/crio-cddc602c5a6f43fa7bd0681009f34d0c400b7bc06604eae92438188d25b63ab1 WatchSource:0}: Error finding container cddc602c5a6f43fa7bd0681009f34d0c400b7bc06604eae92438188d25b63ab1: Status 404 returned error can't find the container with id cddc602c5a6f43fa7bd0681009f34d0c400b7bc06604eae92438188d25b63ab1 Oct 29 23:04:29.341557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.341518 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 23:04:29.341761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.341576 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 23:04:29.851072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.851033 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 29 23:04:29.851261 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.851086 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 29 23:04:29.919812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.919575 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" event={"ID":"23a45c32-e48d-4d51-824a-e9aa89e3f701","Type":"ContainerStarted","Data":"5c49eb5156bfea56d8a36eba792bd7156f2027db722e9c422ce8cfa9e7748b89"} Oct 29 23:04:29.919812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.919644 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" event={"ID":"23a45c32-e48d-4d51-824a-e9aa89e3f701","Type":"ContainerStarted","Data":"cddc602c5a6f43fa7bd0681009f34d0c400b7bc06604eae92438188d25b63ab1"} Oct 29 23:04:29.972723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:29.965240 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" podStartSLOduration=1.965223202 podStartE2EDuration="1.965223202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:29.96473444 +0000 UTC m=+2936.858809363" watchObservedRunningTime="2026-10-29 23:04:29.965223202 +0000 UTC m=+2936.859298124" Oct 29 23:04:30.342226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:30.342080 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 23:04:30.342226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:30.342141 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 23:04:30.849907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:30.849868 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 29 23:04:30.850101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:30.849926 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 29 23:04:30.929409 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:30.929368 5367 generic.go:356] "Generic (PLEG): container finished" podID="23a45c32-e48d-4d51-824a-e9aa89e3f701" containerID="5c49eb5156bfea56d8a36eba792bd7156f2027db722e9c422ce8cfa9e7748b89" exitCode=0 Oct 29 23:04:30.929600 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:30.929506 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" event={"ID":"23a45c32-e48d-4d51-824a-e9aa89e3f701","Type":"ContainerDied","Data":"5c49eb5156bfea56d8a36eba792bd7156f2027db722e9c422ce8cfa9e7748b89"} Oct 29 23:04:31.098214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:31.098172 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:04:31.346394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:31.346369 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:04:31.373722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:31.372926 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:31.373722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:31.373003 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:31.849979 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:31.849943 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 29 23:04:31.850188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:31.850008 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 29 23:04:32.005129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.005088 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:04:32.011094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.010330 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" podUID="04315dfb-bcff-49e6-8f2d-a64659122722" containerName="payload-processing" containerID="cri-o://97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992" gracePeriod=25 Oct 29 23:04:32.018027 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.017279 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" podUID="98ab0a3f-7363-49aa-9ac6-ff1b60437568" containerName="payload-pre-processing" containerID="cri-o://6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095" gracePeriod=25 Oct 29 23:04:32.308605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.307794 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:32.485734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.485662 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca\") pod \"23a45c32-e48d-4d51-824a-e9aa89e3f701\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " Oct 29 23:04:32.486227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.485812 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca\") pod \"23a45c32-e48d-4d51-824a-e9aa89e3f701\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " Oct 29 23:04:32.486227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.485913 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd\" (UniqueName: \"kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd\") pod \"23a45c32-e48d-4d51-824a-e9aa89e3f701\" (UID: \"23a45c32-e48d-4d51-824a-e9aa89e3f701\") " Oct 29 23:04:32.486227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.486122 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca" pod "23a45c32-e48d-4d51-824a-e9aa89e3f701" (UID: "23a45c32-e48d-4d51-824a-e9aa89e3f701"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:32.486502 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.486478 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.488479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.488448 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca" pod "23a45c32-e48d-4d51-824a-e9aa89e3f701" (UID: "23a45c32-e48d-4d51-824a-e9aa89e3f701"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:32.488816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.488684 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd" pod "23a45c32-e48d-4d51-824a-e9aa89e3f701" (UID: "23a45c32-e48d-4d51-824a-e9aa89e3f701"). InnerVolumeSpecName "kube-api-access-ggzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:32.519118 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.519085 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:04:32.542331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.542307 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:04:32.587528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.587497 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw\" (UniqueName: \"kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw\") pod \"04315dfb-bcff-49e6-8f2d-a64659122722\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " Oct 29 23:04:32.587778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.587606 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume\") pod \"04315dfb-bcff-49e6-8f2d-a64659122722\" (UID: \"04315dfb-bcff-49e6-8f2d-a64659122722\") " Oct 29 23:04:32.587778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.587751 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume\") pod \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " Oct 29 23:04:32.588104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588066 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume" pod "04315dfb-bcff-49e6-8f2d-a64659122722" (UID: "04315dfb-bcff-49e6-8f2d-a64659122722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:32.588235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588114 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume" pod "98ab0a3f-7363-49aa-9ac6-ff1b60437568" (UID: "98ab0a3f-7363-49aa-9ac6-ff1b60437568"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:32.588235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk\" (UniqueName: \"kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk\") pod \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\" (UID: \"98ab0a3f-7363-49aa-9ac6-ff1b60437568\") " Oct 29 23:04:32.588751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588677 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a45c32-e48d-4d51-824a-e9aa89e3f701-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.588859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588757 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04315dfb-bcff-49e6-8f2d-a64659122722-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.588859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588771 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzcd\" (UniqueName: \"kubernetes.io/projected/23a45c32-e48d-4d51-824a-e9aa89e3f701-kube-api-access-ggzcd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.588859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.588798 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab0a3f-7363-49aa-9ac6-ff1b60437568-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.590244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.590216 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk" pod "98ab0a3f-7363-49aa-9ac6-ff1b60437568" (UID: "98ab0a3f-7363-49aa-9ac6-ff1b60437568"). InnerVolumeSpecName "kube-api-access-cwjlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:32.590609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.590590 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw" pod "04315dfb-bcff-49e6-8f2d-a64659122722" (UID: "04315dfb-bcff-49e6-8f2d-a64659122722"). InnerVolumeSpecName "kube-api-access-xbqhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:32.689909 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.689826 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjlk\" (UniqueName: \"kubernetes.io/projected/98ab0a3f-7363-49aa-9ac6-ff1b60437568-kube-api-access-cwjlk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.689909 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.689858 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqhw\" (UniqueName: \"kubernetes.io/projected/04315dfb-bcff-49e6-8f2d-a64659122722-kube-api-access-xbqhw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.855161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.855131 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:04:32.855469 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.855448 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:04:32.856238 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.856210 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:04:32.996064 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:32.996029 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 23:04:33.001100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.001080 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" Oct 29 23:04:33.001100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.001088 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24" event={"ID":"23a45c32-e48d-4d51-824a-e9aa89e3f701","Type":"ContainerDied","Data":"cddc602c5a6f43fa7bd0681009f34d0c400b7bc06604eae92438188d25b63ab1"} Oct 29 23:04:33.001283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.001140 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddc602c5a6f43fa7bd0681009f34d0c400b7bc06604eae92438188d25b63ab1" Oct 29 23:04:33.002813 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.002787 5367 generic.go:356] "Generic (PLEG): container finished" podID="251fc4be-6212-4e18-9759-8802ee120fd0" containerID="4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa" exitCode=0 Oct 29 23:04:33.002949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.002831 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" Oct 29 23:04:33.002949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.002870 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" event={"ID":"251fc4be-6212-4e18-9759-8802ee120fd0","Type":"ContainerDied","Data":"4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa"} Oct 29 23:04:33.002949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.002915 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv" event={"ID":"251fc4be-6212-4e18-9759-8802ee120fd0","Type":"ContainerDied","Data":"a170ba110c1a66d6f50e45e4f001608d2311f36a64b94bf9589ec5539d502f78"} Oct 29 23:04:33.002949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.002944 5367 scope.go:122] "RemoveContainer" containerID="4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa" Oct 29 23:04:33.004920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.004743 5367 generic.go:356] "Generic (PLEG): container finished" podID="98ab0a3f-7363-49aa-9ac6-ff1b60437568" containerID="6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095" exitCode=0 Oct 29 23:04:33.004920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.004815 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" Oct 29 23:04:33.004920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.004848 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" event={"ID":"98ab0a3f-7363-49aa-9ac6-ff1b60437568","Type":"ContainerDied","Data":"6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095"} Oct 29 23:04:33.004920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.004884 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc" event={"ID":"98ab0a3f-7363-49aa-9ac6-ff1b60437568","Type":"ContainerDied","Data":"539438f44d71c9c2710fdfd9046ad37e283d8bf6dc3c196841ddd21023a0e488"} Oct 29 23:04:33.017158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.017050 5367 generic.go:356] "Generic (PLEG): container finished" podID="04315dfb-bcff-49e6-8f2d-a64659122722" containerID="97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992" exitCode=0 Oct 29 23:04:33.017650 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.017626 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" event={"ID":"04315dfb-bcff-49e6-8f2d-a64659122722","Type":"ContainerDied","Data":"97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992"} Oct 29 23:04:33.017944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.017917 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" event={"ID":"04315dfb-bcff-49e6-8f2d-a64659122722","Type":"ContainerDied","Data":"20f0572c398122777832a300899abd663db6eaebb4c55432a467933135e1d737"} Oct 29 23:04:33.018060 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.017762 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6" Oct 29 23:04:33.018060 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.017690 5367 scope.go:122] "RemoveContainer" containerID="4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa" Oct 29 23:04:33.019384 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:04:33.019358 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa\": container with ID starting with 4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa not found: ID does not exist" containerID="4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa" Oct 29 23:04:33.019768 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.019739 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa"} err="failed to get container status \"4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa\": rpc error: code = NotFound desc = could not find container \"4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa\": container with ID starting with 4b61558d229d3f96259f4dd9d828f3288b123fe30f0074198e6a92326f0056aa not found: ID does not exist" Oct 29 23:04:33.019939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.019924 5367 scope.go:122] "RemoveContainer" containerID="6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095" Oct 29 23:04:33.034949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.034932 5367 scope.go:122] "RemoveContainer" containerID="6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095" Oct 29 23:04:33.035518 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:04:33.035380 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095\": container with ID starting with 6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095 not found: ID does not exist" containerID="6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095" Oct 29 23:04:33.035518 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.035420 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095"} err="failed to get container status \"6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095\": rpc error: code = NotFound desc = could not find container \"6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095\": container with ID starting with 6e837a00d18cf81f2e3465226a334fad415c6d32056a78750bef807595af0095 not found: ID does not exist" Oct 29 23:04:33.035518 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.035445 5367 scope.go:122] "RemoveContainer" containerID="97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992" Oct 29 23:04:33.050215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.050192 5367 scope.go:122] "RemoveContainer" containerID="97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992" Oct 29 23:04:33.050598 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:04:33.050573 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992\": container with ID starting with 97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992 not found: ID does not exist" containerID="97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992" Oct 29 23:04:33.050740 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.050716 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992"} err="failed to get container status \"97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992\": rpc error: code = NotFound desc = could not find container \"97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992\": container with ID starting with 97c909c9468dfe1560b46c945380be6a879d1375b58a8e9fad300c711a310992 not found: ID does not exist" Oct 29 23:04:33.095184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095150 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095210 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095247 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095320 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095351 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095508 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095614 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095593 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095753 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095725 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095772 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs\") pod \"251fc4be-6212-4e18-9759-8802ee120fd0\" (UID: \"251fc4be-6212-4e18-9759-8802ee120fd0\") " Oct 29 23:04:33.095824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095806 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.095991 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.095966 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.096172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.096150 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:33.097457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.096599 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.097457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.096934 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fc4be-6212-4e18-9759-8802ee120fd0-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.097457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.097232 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.097457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.097250 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.097931 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.097907 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.098322 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.098301 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:33.098759 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.098739 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.099029 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.099009 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "kube-api-access-wxrgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.099454 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.099429 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy" pod "251fc4be-6212-4e18-9759-8802ee120fd0" (UID: "251fc4be-6212-4e18-9759-8802ee120fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.132252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.132226 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc"] Oct 29 23:04:33.136038 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.136014 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-9ddb46db7-n4spc"] Oct 29 23:04:33.190489 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.190459 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6"] Oct 29 23:04:33.194230 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.194204 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-66bc479ffd-9qht6"] Oct 29 23:04:33.199040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.199018 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fc4be-6212-4e18-9759-8802ee120fd0-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.199040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.199038 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.199194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.199047 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.199194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.199055 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.199194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.199069 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrgw\" (UniqueName: \"kubernetes.io/projected/251fc4be-6212-4e18-9759-8802ee120fd0-kube-api-access-wxrgw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.199194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.199082 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fc4be-6212-4e18-9759-8802ee120fd0-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.224462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.224438 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04315dfb-bcff-49e6-8f2d-a64659122722" path="/var/lib/kubelet/pods/04315dfb-bcff-49e6-8f2d-a64659122722/volumes" Oct 29 23:04:33.224883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.224868 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ab0a3f-7363-49aa-9ac6-ff1b60437568" path="/var/lib/kubelet/pods/98ab0a3f-7363-49aa-9ac6-ff1b60437568/volumes" Oct 29 23:04:33.351557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.351528 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv"] Oct 29 23:04:33.355293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.355264 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-rjwzv"] Oct 29 23:04:33.779792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:33.779757 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld"] Oct 29 23:04:35.226488 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:35.226453 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" path="/var/lib/kubelet/pods/251fc4be-6212-4e18-9759-8802ee120fd0/volumes" Oct 29 23:04:38.780357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:38.780303 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" podUID="7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" containerName="payload-pre-processing" containerID="cri-o://696f724e6b83de2b23ba4b8f4343fbbe2f88983fba72bff8d460200abd5bfc6d" gracePeriod=25 Oct 29 23:04:38.856976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:38.856943 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2"] Oct 29 23:04:39.106365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.106336 5367 generic.go:356] "Generic (PLEG): container finished" podID="7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" containerID="696f724e6b83de2b23ba4b8f4343fbbe2f88983fba72bff8d460200abd5bfc6d" exitCode=0 Oct 29 23:04:39.106548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.106406 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" event={"ID":"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee","Type":"ContainerDied","Data":"696f724e6b83de2b23ba4b8f4343fbbe2f88983fba72bff8d460200abd5bfc6d"} Oct 29 23:04:39.128434 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.128411 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:04:39.200456 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.200424 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume\") pod \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " Oct 29 23:04:39.200636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.200548 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr\" (UniqueName: \"kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr\") pod \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\" (UID: \"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee\") " Oct 29 23:04:39.200873 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.200851 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume" pod "7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" (UID: "7363cfa8-d323-4ebd-a3c6-44497d2ff1ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:39.201194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.201167 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:39.202922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.202897 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr" pod "7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" (UID: "7363cfa8-d323-4ebd-a3c6-44497d2ff1ee"). InnerVolumeSpecName "kube-api-access-64gnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:39.212264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.212238 5367 scope.go:122] "RemoveContainer" containerID="fd1b438a32f7400b0a131fc078402f09cfb159689024dc1b02081f3185d55273" Oct 29 23:04:39.225859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.225835 5367 scope.go:122] "RemoveContainer" containerID="30c4e4b4a8579dfd59724d8b0de4b45d192ef9a13994d3d337a1a761e28b6939" Oct 29 23:04:39.236598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.236576 5367 scope.go:122] "RemoveContainer" containerID="62e7e2d56b320855a2c7c01a0e8db2d0357c5ea6db1c9ad80a1fec3769556bba" Oct 29 23:04:39.302670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:39.302639 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64gnr\" (UniqueName: \"kubernetes.io/projected/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee-kube-api-access-64gnr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:40.115832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:40.115804 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" Oct 29 23:04:40.116295 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:40.115841 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld" event={"ID":"7363cfa8-d323-4ebd-a3c6-44497d2ff1ee","Type":"ContainerDied","Data":"e206d61178d595f98d7dbc854f7822ebcb9b51284b8c883be5caf0ec7ebce068"} Oct 29 23:04:40.116295 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:40.115906 5367 scope.go:122] "RemoveContainer" containerID="696f724e6b83de2b23ba4b8f4343fbbe2f88983fba72bff8d460200abd5bfc6d" Oct 29 23:04:40.178523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:40.178492 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld"] Oct 29 23:04:40.191681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:40.191643 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-b45d997f-vsvld"] Oct 29 23:04:41.223240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:41.223212 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" path="/var/lib/kubelet/pods/7363cfa8-d323-4ebd-a3c6-44497d2ff1ee/volumes" Oct 29 23:04:42.977533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:42.977500 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt"] Oct 29 23:04:42.977965 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:42.977762 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" containerID="cri-o://868f97a08ff31a632a0d4dc8ebf32d10bbc64a96c7be9db4281147f5fba1c1d9" gracePeriod=30 Oct 29 23:04:43.480300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.480254 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff"] Oct 29 23:04:43.480955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.480932 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" containerName="payload-pre-processing" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.480955 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" containerName="payload-pre-processing" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.480985 5367 cpu_manager.go:414] "containerMap: removing container" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.480992 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481056 5367 cpu_manager.go:414] "containerMap: removing container" podUID="98ab0a3f-7363-49aa-9ac6-ff1b60437568" containerName="payload-pre-processing" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481064 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ab0a3f-7363-49aa-9ac6-ff1b60437568" containerName="payload-pre-processing" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481079 5367 cpu_manager.go:414] "containerMap: removing container" podUID="23a45c32-e48d-4d51-824a-e9aa89e3f701" containerName="revoke-keys" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481086 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a45c32-e48d-4d51-824a-e9aa89e3f701" containerName="revoke-keys" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481100 5367 cpu_manager.go:414] "containerMap: removing container" podUID="04315dfb-bcff-49e6-8f2d-a64659122722" containerName="payload-processing" Oct 29 23:04:43.481107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481108 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04315dfb-bcff-49e6-8f2d-a64659122722" containerName="payload-processing" Oct 29 23:04:43.481560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481253 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7363cfa8-d323-4ebd-a3c6-44497d2ff1ee" containerName="payload-pre-processing" Oct 29 23:04:43.481560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481273 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="251fc4be-6212-4e18-9759-8802ee120fd0" containerName="istio-proxy" Oct 29 23:04:43.481560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481292 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ab0a3f-7363-49aa-9ac6-ff1b60437568" containerName="payload-pre-processing" Oct 29 23:04:43.481560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481304 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a45c32-e48d-4d51-824a-e9aa89e3f701" containerName="revoke-keys" Oct 29 23:04:43.481560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.481315 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="04315dfb-bcff-49e6-8f2d-a64659122722" containerName="payload-processing" Oct 29 23:04:43.484196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.484175 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.504270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.504247 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff"] Oct 29 23:04:43.581466 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.581429 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:43.581653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.581544 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:04:43.592934 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.592910 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.593055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.593035 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.593117 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.593074 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.694066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.694035 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.694188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.694160 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.694245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.694202 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.694918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.694898 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.696466 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.696445 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.715451 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.715431 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb\") pod \"maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.797446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.797381 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:43.857394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:43.857360 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerName="payload-pre-processing" containerID="cri-o://22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2" gracePeriod=25 Oct 29 23:04:44.123820 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:04:44.123785 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee46a346_9957_482b_a78f_2b7d64f1b41f.slice/crio-6374ec378918186055ce779ddd33ba6a9fa90a1b2c3f08e08bd0c7457600a2f1 WatchSource:0}: Error finding container 6374ec378918186055ce779ddd33ba6a9fa90a1b2c3f08e08bd0c7457600a2f1: Status 404 returned error can't find the container with id 6374ec378918186055ce779ddd33ba6a9fa90a1b2c3f08e08bd0c7457600a2f1 Oct 29 23:04:44.124266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:44.123960 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff"] Oct 29 23:04:44.168758 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:44.168728 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" event={"ID":"ee46a346-9957-482b-a78f-2b7d64f1b41f","Type":"ContainerStarted","Data":"6374ec378918186055ce779ddd33ba6a9fa90a1b2c3f08e08bd0c7457600a2f1"} Oct 29 23:04:45.178271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:45.178230 5367 generic.go:356] "Generic (PLEG): container finished" podID="ee46a346-9957-482b-a78f-2b7d64f1b41f" containerID="1b4e7a33c8fc679fea7e291716c204fed2cbe1f885ccd2617cbad8da9dec6ff2" exitCode=0 Oct 29 23:04:45.178670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:45.178311 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" event={"ID":"ee46a346-9957-482b-a78f-2b7d64f1b41f","Type":"ContainerDied","Data":"1b4e7a33c8fc679fea7e291716c204fed2cbe1f885ccd2617cbad8da9dec6ff2"} Oct 29 23:04:46.352477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.352453 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:46.535141 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.535040 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca\") pod \"ee46a346-9957-482b-a78f-2b7d64f1b41f\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " Oct 29 23:04:46.535141 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.535126 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca\") pod \"ee46a346-9957-482b-a78f-2b7d64f1b41f\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " Oct 29 23:04:46.535424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.535382 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb\") pod \"ee46a346-9957-482b-a78f-2b7d64f1b41f\" (UID: \"ee46a346-9957-482b-a78f-2b7d64f1b41f\") " Oct 29 23:04:46.535492 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.535451 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca" pod "ee46a346-9957-482b-a78f-2b7d64f1b41f" (UID: "ee46a346-9957-482b-a78f-2b7d64f1b41f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:46.535837 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.535815 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.537380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.537357 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca" pod "ee46a346-9957-482b-a78f-2b7d64f1b41f" (UID: "ee46a346-9957-482b-a78f-2b7d64f1b41f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:46.537639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.537616 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb" pod "ee46a346-9957-482b-a78f-2b7d64f1b41f" (UID: "ee46a346-9957-482b-a78f-2b7d64f1b41f"). InnerVolumeSpecName "kube-api-access-sjrtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:46.637570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.637541 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/ee46a346-9957-482b-a78f-2b7d64f1b41f-kube-api-access-sjrtb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.637570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:46.637568 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee46a346-9957-482b-a78f-2b7d64f1b41f-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.217998 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:47.217962 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" Oct 29 23:04:47.227512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:47.227460 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff" event={"ID":"ee46a346-9957-482b-a78f-2b7d64f1b41f","Type":"ContainerDied","Data":"6374ec378918186055ce779ddd33ba6a9fa90a1b2c3f08e08bd0c7457600a2f1"} Oct 29 23:04:47.227681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:47.227535 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6374ec378918186055ce779ddd33ba6a9fa90a1b2c3f08e08bd0c7457600a2f1" Oct 29 23:04:48.232533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.232506 5367 generic.go:356] "Generic (PLEG): container finished" podID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerID="868f97a08ff31a632a0d4dc8ebf32d10bbc64a96c7be9db4281147f5fba1c1d9" exitCode=0 Oct 29 23:04:48.232922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.232587 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" event={"ID":"0229cbbd-cd97-4074-9d0c-b07aaf7063ba","Type":"ContainerDied","Data":"868f97a08ff31a632a0d4dc8ebf32d10bbc64a96c7be9db4281147f5fba1c1d9"} Oct 29 23:04:48.232922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.232655 5367 scope.go:122] "RemoveContainer" containerID="a6ffe6d5198cba2c077811b86643645434dad68d4c6c649de56bbfccaffe1096" Oct 29 23:04:48.262049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.262031 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 23:04:48.361864 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.361831 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.361968 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362020 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362059 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362097 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362151 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362254 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362350 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362318 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.362432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.362349 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data\") pod \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\" (UID: \"0229cbbd-cd97-4074-9d0c-b07aaf7063ba\") " Oct 29 23:04:48.363054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.363021 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.363634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.363245 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.363634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.363609 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.363634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.363632 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.363921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.363777 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:48.364069 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.364039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.364182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.364110 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.365176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.365146 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:48.365282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.365234 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "kube-api-access-jzhrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:48.365755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.365729 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:48.366400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.366373 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy" pod "0229cbbd-cd97-4074-9d0c-b07aaf7063ba" (UID: "0229cbbd-cd97-4074-9d0c-b07aaf7063ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.465175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465139 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.465175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465171 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.465175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465180 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.465422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465190 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhrv\" (UniqueName: \"kubernetes.io/projected/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-kube-api-access-jzhrv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.465422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465200 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.465422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465208 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.465422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:48.465216 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0229cbbd-cd97-4074-9d0c-b07aaf7063ba-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:04:49.251927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:49.251893 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" Oct 29 23:04:49.251927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:49.251905 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt" event={"ID":"0229cbbd-cd97-4074-9d0c-b07aaf7063ba","Type":"ContainerDied","Data":"31b6fb056069bf040cf6d14de791a46a8ece2f22c1e5bd5c70857b68d9b78253"} Oct 29 23:04:49.252501 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:49.251996 5367 scope.go:122] "RemoveContainer" containerID="868f97a08ff31a632a0d4dc8ebf32d10bbc64a96c7be9db4281147f5fba1c1d9" Oct 29 23:04:49.384353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:49.384321 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt"] Oct 29 23:04:49.394572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:49.394541 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-f7vwt"] Oct 29 23:04:51.221969 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:51.221931 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" path="/var/lib/kubelet/pods/0229cbbd-cd97-4074-9d0c-b07aaf7063ba/volumes" Oct 29 23:04:52.331306 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:04:52.331268 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:04:52.825129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:52.825092 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:04:58.030079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:58.030045 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9"] Oct 29 23:04:58.043719 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:58.043672 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr"] Oct 29 23:04:58.047120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:58.047099 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-115d83cc-fbll9"] Oct 29 23:04:58.049964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:58.049944 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c7700b-6z7lr"] Oct 29 23:04:58.176823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:58.176787 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24"] Oct 29 23:04:58.180104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:58.180077 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dbb507-skl24"] Oct 29 23:04:59.037906 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.037834 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z"] Oct 29 23:04:59.048209 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.048139 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef262b67-qlf6z"] Oct 29 23:04:59.223502 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.223472 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a45c32-e48d-4d51-824a-e9aa89e3f701" path="/var/lib/kubelet/pods/23a45c32-e48d-4d51-824a-e9aa89e3f701/volumes" Oct 29 23:04:59.223997 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.223981 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d5a7bd-378b-4f2e-b596-eed156b70bb0" path="/var/lib/kubelet/pods/31d5a7bd-378b-4f2e-b596-eed156b70bb0/volumes" Oct 29 23:04:59.224376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.224363 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f49f61-a708-4b72-bb22-09b6662b7979" path="/var/lib/kubelet/pods/93f49f61-a708-4b72-bb22-09b6662b7979/volumes" Oct 29 23:04:59.225134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.225115 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69f26a2-cc87-409b-9a95-0b2f44910015" path="/var/lib/kubelet/pods/e69f26a2-cc87-409b-9a95-0b2f44910015/volumes" Oct 29 23:04:59.922648 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.922612 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5"] Oct 29 23:04:59.923615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923592 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" Oct 29 23:04:59.923787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923619 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" Oct 29 23:04:59.923787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923630 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ee46a346-9957-482b-a78f-2b7d64f1b41f" containerName="revoke-keys" Oct 29 23:04:59.923787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923638 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee46a346-9957-482b-a78f-2b7d64f1b41f" containerName="revoke-keys" Oct 29 23:04:59.923787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923680 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" Oct 29 23:04:59.923787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923689 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" Oct 29 23:04:59.924094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923885 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" Oct 29 23:04:59.924094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923908 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee46a346-9957-482b-a78f-2b7d64f1b41f" containerName="revoke-keys" Oct 29 23:04:59.924094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.923933 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0229cbbd-cd97-4074-9d0c-b07aaf7063ba" containerName="istio-proxy" Oct 29 23:04:59.937790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.937759 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5"] Oct 29 23:04:59.938205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.938183 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:04:59.957251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.957227 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c74e55\"" Oct 29 23:04:59.997425 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:04:59.997392 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c74e55-dockercfg-c47lc\"" Oct 29 23:05:00.041829 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.041794 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.042298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.041918 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vfm\" (UniqueName: \"kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.143597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.143554 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.143848 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.143643 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vfm\" (UniqueName: \"kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.144380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.144358 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.168570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.168529 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vfm\" (UniqueName: \"kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm\") pod \"payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.255605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.255514 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:00.565205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:00.565175 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5"] Oct 29 23:05:00.565972 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:00.565947 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612a9ec7_d897_49a0_b224_aeae8495961d.slice/crio-ab49bffd518edd181fdc943bcf38b321aa11fad50bb21247aeb15ab9c5e571d7 WatchSource:0}: Error finding container ab49bffd518edd181fdc943bcf38b321aa11fad50bb21247aeb15ab9c5e571d7: Status 404 returned error can't find the container with id ab49bffd518edd181fdc943bcf38b321aa11fad50bb21247aeb15ab9c5e571d7 Oct 29 23:05:01.388272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:01.388232 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" event={"ID":"612a9ec7-d897-49a0-b224-aeae8495961d","Type":"ContainerStarted","Data":"26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7"} Oct 29 23:05:01.388733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:01.388303 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" event={"ID":"612a9ec7-d897-49a0-b224-aeae8495961d","Type":"ContainerStarted","Data":"ab49bffd518edd181fdc943bcf38b321aa11fad50bb21247aeb15ab9c5e571d7"} Oct 29 23:05:01.388733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:01.388461 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:01.434511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:01.434445 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" podStartSLOduration=2.434429577 podStartE2EDuration="2.434429577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:01.432258783 +0000 UTC m=+2968.326333703" watchObservedRunningTime="2026-10-29 23:05:01.434429577 +0000 UTC m=+2968.328504493" Oct 29 23:05:02.825266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:02.825226 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:05:04.020951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.020926 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 23:05:04.106439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.106394 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb\" (UniqueName: \"kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb\") pod \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " Oct 29 23:05:04.106659 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.106482 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume\") pod \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\" (UID: \"a1583b34-c0be-45ae-b6cf-b53ba9f40258\") " Oct 29 23:05:04.106948 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.106920 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume" pod "a1583b34-c0be-45ae-b6cf-b53ba9f40258" (UID: "a1583b34-c0be-45ae-b6cf-b53ba9f40258"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:04.107157 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.107139 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1583b34-c0be-45ae-b6cf-b53ba9f40258-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.109219 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.109197 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb" pod "a1583b34-c0be-45ae-b6cf-b53ba9f40258" (UID: "a1583b34-c0be-45ae-b6cf-b53ba9f40258"). InnerVolumeSpecName "kube-api-access-h8dlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:04.208528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.208497 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8dlb\" (UniqueName: \"kubernetes.io/projected/a1583b34-c0be-45ae-b6cf-b53ba9f40258-kube-api-access-h8dlb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.413598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.413520 5367 generic.go:356] "Generic (PLEG): container finished" podID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerID="22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2" exitCode=0 Oct 29 23:05:04.413598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.413568 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" Oct 29 23:05:04.413822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.413600 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" event={"ID":"a1583b34-c0be-45ae-b6cf-b53ba9f40258","Type":"ContainerDied","Data":"22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2"} Oct 29 23:05:04.413822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.413654 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2" event={"ID":"a1583b34-c0be-45ae-b6cf-b53ba9f40258","Type":"ContainerDied","Data":"0c68ea42c10b2398e49974a5c600f335df49e35774c3bf969d1887b82d27f5d8"} Oct 29 23:05:04.413822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.413676 5367 scope.go:122] "RemoveContainer" containerID="22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2" Oct 29 23:05:04.424679 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.424661 5367 scope.go:122] "RemoveContainer" containerID="22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2" Oct 29 23:05:04.424943 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:04.424924 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2\": container with ID starting with 22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2 not found: ID does not exist" containerID="22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2" Oct 29 23:05:04.424997 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.424952 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2"} err="failed to get container status \"22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2\": rpc error: code = NotFound desc = could not find container \"22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2\": container with ID starting with 22af8e542b928f29bc4e7bba67c1eb00c97c8061c75ec8e69660b6e43f0661c2 not found: ID does not exist" Oct 29 23:05:04.472286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.472261 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2"] Oct 29 23:05:04.479572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.479547 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dbb507-59876dbb59-jfcl2"] Oct 29 23:05:04.502506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.502481 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb"] Oct 29 23:05:04.503055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.503041 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerName="payload-pre-processing" Oct 29 23:05:04.503104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.503056 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerName="payload-pre-processing" Oct 29 23:05:04.503194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.503185 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" containerName="payload-pre-processing" Oct 29 23:05:04.507653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.507639 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.524368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.524344 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:04.524525 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.524505 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:05:04.527529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.527508 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb"] Oct 29 23:05:04.612597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.612566 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5fr\" (UniqueName: \"kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.612785 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.612613 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.612839 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.612782 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.714204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.714125 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5fr\" (UniqueName: \"kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.714204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.714192 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.714396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.714332 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.715121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.715091 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.716678 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.716657 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.735414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.735390 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5fr\" (UniqueName: \"kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:04.820653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:04.820620 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:05.140635 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:05.139801 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09607296_b9c5_405a_b307_195224c0d1fb.slice/crio-457f05c17bebfd5f5487b112066d9295ad1d76cc5b45a595432d54bcf0781009 WatchSource:0}: Error finding container 457f05c17bebfd5f5487b112066d9295ad1d76cc5b45a595432d54bcf0781009: Status 404 returned error can't find the container with id 457f05c17bebfd5f5487b112066d9295ad1d76cc5b45a595432d54bcf0781009 Oct 29 23:05:05.146139 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.146108 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb"] Oct 29 23:05:05.222308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.222279 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1583b34-c0be-45ae-b6cf-b53ba9f40258" path="/var/lib/kubelet/pods/a1583b34-c0be-45ae-b6cf-b53ba9f40258/volumes" Oct 29 23:05:05.422806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.422728 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" event={"ID":"09607296-b9c5-405a-b307-195224c0d1fb","Type":"ContainerStarted","Data":"457f05c17bebfd5f5487b112066d9295ad1d76cc5b45a595432d54bcf0781009"} Oct 29 23:05:05.838112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.838077 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k"] Oct 29 23:05:05.842370 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.842345 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:05.863217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.863192 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k"] Oct 29 23:05:05.933890 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.933862 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:05.934027 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.933995 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntlq\" (UniqueName: \"kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:05.934126 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:05.934112 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.035862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.035831 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntlq\" (UniqueName: \"kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.036043 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.035934 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.036043 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.035998 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.036688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.036664 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.038455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.038434 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.056422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.056391 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntlq\" (UniqueName: \"kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq\") pod \"maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.155610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.155578 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:06.433900 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.433821 5367 generic.go:356] "Generic (PLEG): container finished" podID="09607296-b9c5-405a-b307-195224c0d1fb" containerID="f5b26e88e9f6ab7e64c5ce1fe97dc23c85dd9b95300e76028539b32c30acfc7c" exitCode=0 Oct 29 23:05:06.433900 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.433875 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" event={"ID":"09607296-b9c5-405a-b307-195224c0d1fb","Type":"ContainerDied","Data":"f5b26e88e9f6ab7e64c5ce1fe97dc23c85dd9b95300e76028539b32c30acfc7c"} Oct 29 23:05:06.457021 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:06.456990 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k"] Oct 29 23:05:06.457223 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:06.457203 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71c5ec8_8dff_44ca_9ec1_c98d0d34678f.slice/crio-fe7f9b65aa08c02ce286d97e27bc34b2662dc360da339395e2e208dc0d93df0c WatchSource:0}: Error finding container fe7f9b65aa08c02ce286d97e27bc34b2662dc360da339395e2e208dc0d93df0c: Status 404 returned error can't find the container with id fe7f9b65aa08c02ce286d97e27bc34b2662dc360da339395e2e208dc0d93df0c Oct 29 23:05:07.453886 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.453849 5367 generic.go:356] "Generic (PLEG): container finished" podID="d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" containerID="18ee0d3d79d3ac9d40547c6fc182f62e2745b0e2b3a84b6a575d363ebea9d094" exitCode=0 Oct 29 23:05:07.454242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.453931 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" event={"ID":"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f","Type":"ContainerDied","Data":"18ee0d3d79d3ac9d40547c6fc182f62e2745b0e2b3a84b6a575d363ebea9d094"} Oct 29 23:05:07.454242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.454006 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" event={"ID":"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f","Type":"ContainerStarted","Data":"fe7f9b65aa08c02ce286d97e27bc34b2662dc360da339395e2e208dc0d93df0c"} Oct 29 23:05:07.604543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.604518 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:07.757863 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.757782 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca\") pod \"09607296-b9c5-405a-b307-195224c0d1fb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " Oct 29 23:05:07.758032 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.757911 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca\") pod \"09607296-b9c5-405a-b307-195224c0d1fb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " Oct 29 23:05:07.758032 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.758012 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr\" (UniqueName: \"kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr\") pod \"09607296-b9c5-405a-b307-195224c0d1fb\" (UID: \"09607296-b9c5-405a-b307-195224c0d1fb\") " Oct 29 23:05:07.758210 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.758185 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca" pod "09607296-b9c5-405a-b307-195224c0d1fb" (UID: "09607296-b9c5-405a-b307-195224c0d1fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:07.758529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.758508 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.760037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.760016 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca" pod "09607296-b9c5-405a-b307-195224c0d1fb" (UID: "09607296-b9c5-405a-b307-195224c0d1fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:07.760156 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.760137 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr" pod "09607296-b9c5-405a-b307-195224c0d1fb" (UID: "09607296-b9c5-405a-b307-195224c0d1fb"). InnerVolumeSpecName "kube-api-access-ws5fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:07.859680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.859630 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09607296-b9c5-405a-b307-195224c0d1fb-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.859680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:07.859660 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws5fr\" (UniqueName: \"kubernetes.io/projected/09607296-b9c5-405a-b307-195224c0d1fb-kube-api-access-ws5fr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:08.463533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.463500 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" Oct 29 23:05:08.463960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.463532 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb" event={"ID":"09607296-b9c5-405a-b307-195224c0d1fb","Type":"ContainerDied","Data":"457f05c17bebfd5f5487b112066d9295ad1d76cc5b45a595432d54bcf0781009"} Oct 29 23:05:08.463960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.463584 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457f05c17bebfd5f5487b112066d9295ad1d76cc5b45a595432d54bcf0781009" Oct 29 23:05:08.641153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.641108 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:08.698837 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.698810 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca\") pod \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " Oct 29 23:05:08.699024 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.698870 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq\" (UniqueName: \"kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq\") pod \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " Oct 29 23:05:08.699091 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.699050 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca\") pod \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\" (UID: \"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f\") " Oct 29 23:05:08.699226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.699206 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca" pod "d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" (UID: "d71c5ec8-8dff-44ca-9ec1-c98d0d34678f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:08.699567 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.699544 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:08.701186 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.701158 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq" pod "d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" (UID: "d71c5ec8-8dff-44ca-9ec1-c98d0d34678f"). InnerVolumeSpecName "kube-api-access-jntlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:08.701287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.701214 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca" pod "d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" (UID: "d71c5ec8-8dff-44ca-9ec1-c98d0d34678f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:08.801303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.801222 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:08.801303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:08.801253 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jntlq\" (UniqueName: \"kubernetes.io/projected/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f-kube-api-access-jntlq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.352728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:09.352678 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff"] Oct 29 23:05:09.364321 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:09.364295 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0906f4-qg9ff"] Oct 29 23:05:09.472941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:09.472913 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" Oct 29 23:05:09.472941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:09.472942 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k" event={"ID":"d71c5ec8-8dff-44ca-9ec1-c98d0d34678f","Type":"ContainerDied","Data":"fe7f9b65aa08c02ce286d97e27bc34b2662dc360da339395e2e208dc0d93df0c"} Oct 29 23:05:09.473429 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:09.472984 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7f9b65aa08c02ce286d97e27bc34b2662dc360da339395e2e208dc0d93df0c" Oct 29 23:05:10.030009 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:10.029973 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl"] Oct 29 23:05:10.043476 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:10.043444 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68befb42-sqfwl"] Oct 29 23:05:10.636042 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:10.635894 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7"] Oct 29 23:05:11.230253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:11.230214 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a5c283-414a-4b5f-84b2-fd0c7f071617" path="/var/lib/kubelet/pods/46a5c283-414a-4b5f-84b2-fd0c7f071617/volumes" Oct 29 23:05:11.230758 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:11.230743 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee46a346-9957-482b-a78f-2b7d64f1b41f" path="/var/lib/kubelet/pods/ee46a346-9957-482b-a78f-2b7d64f1b41f/volumes" Oct 29 23:05:12.417327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:12.417298 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:16.168099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.168015 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5"] Oct 29 23:05:16.168958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.168913 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" containerName="revoke-keys" Oct 29 23:05:16.168958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.168939 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" containerName="revoke-keys" Oct 29 23:05:16.169253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.168986 5367 cpu_manager.go:414] "containerMap: removing container" podUID="09607296-b9c5-405a-b307-195224c0d1fb" containerName="revoke-keys" Oct 29 23:05:16.169253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.168996 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09607296-b9c5-405a-b307-195224c0d1fb" containerName="revoke-keys" Oct 29 23:05:16.169253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.169138 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" containerName="revoke-keys" Oct 29 23:05:16.169253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.169158 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="09607296-b9c5-405a-b307-195224c0d1fb" containerName="revoke-keys" Oct 29 23:05:16.174411 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.174389 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.195036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.195008 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:16.197307 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.197283 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:05:16.200877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.200853 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5"] Oct 29 23:05:16.210011 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.209986 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp"] Oct 29 23:05:16.210245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.210219 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" containerID="cri-o://9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099" gracePeriod=30 Oct 29 23:05:16.224154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.224129 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64w6j\" (UniqueName: \"kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.224278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.224220 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.224367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.224290 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.348167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.348127 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64w6j\" (UniqueName: \"kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.348380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.348267 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.348380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.348362 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.359727 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.349328 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.378228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.377511 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d"] Oct 29 23:05:16.394726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.380521 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.394726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.384212 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64w6j\" (UniqueName: \"kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j\") pod \"maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.414195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.413495 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb"] Oct 29 23:05:16.498589 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.498520 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:16.717776 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.717739 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf"] Oct 29 23:05:16.723256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.723235 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.741810 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.741787 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf"] Oct 29 23:05:16.780683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.780623 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.780683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.780667 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz87p\" (UniqueName: \"kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.780939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.780924 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.804174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.804146 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5"] Oct 29 23:05:16.804359 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:16.804334 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670ca2f2_0a4e_477c_a707_f727fbcf9dc6.slice/crio-0e1bddfaf2a9da832628d7c2de9bfe0f69e8206e66cce576cc59c21e29254c95 WatchSource:0}: Error finding container 0e1bddfaf2a9da832628d7c2de9bfe0f69e8206e66cce576cc59c21e29254c95: Status 404 returned error can't find the container with id 0e1bddfaf2a9da832628d7c2de9bfe0f69e8206e66cce576cc59c21e29254c95 Oct 29 23:05:16.882321 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.882290 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.882484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.882359 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.882484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.882380 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz87p\" (UniqueName: \"kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.883074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.883052 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.884530 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.884513 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:16.906314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:16.906290 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz87p\" (UniqueName: \"kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:17.053757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:17.048768 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:17.397856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:17.397827 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf"] Oct 29 23:05:17.398870 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:17.398841 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15205b7_2c11_45b1_8f4d_431f4c38c8e4.slice/crio-b99ce9d8605d1c0f0dd4290d8baad91b7f49a2b21ae75c55165e76dd89dbc59c WatchSource:0}: Error finding container b99ce9d8605d1c0f0dd4290d8baad91b7f49a2b21ae75c55165e76dd89dbc59c: Status 404 returned error can't find the container with id b99ce9d8605d1c0f0dd4290d8baad91b7f49a2b21ae75c55165e76dd89dbc59c Oct 29 23:05:17.540393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:17.540346 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" event={"ID":"670ca2f2-0a4e-477c-a707-f727fbcf9dc6","Type":"ContainerStarted","Data":"94a8024f8ad997263ab790e1833f9af045568776b3dd972ce38cf9a638aa028c"} Oct 29 23:05:17.540592 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:17.540409 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" event={"ID":"670ca2f2-0a4e-477c-a707-f727fbcf9dc6","Type":"ContainerStarted","Data":"0e1bddfaf2a9da832628d7c2de9bfe0f69e8206e66cce576cc59c21e29254c95"} Oct 29 23:05:17.542155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:17.542127 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" event={"ID":"f15205b7-2c11-45b1-8f4d-431f4c38c8e4","Type":"ContainerStarted","Data":"b99ce9d8605d1c0f0dd4290d8baad91b7f49a2b21ae75c55165e76dd89dbc59c"} Oct 29 23:05:17.589015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:17.588964 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" podStartSLOduration=1.588949919 podStartE2EDuration="1.588949919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:17.585163648 +0000 UTC m=+2984.479238570" watchObservedRunningTime="2026-10-29 23:05:17.588949919 +0000 UTC m=+2984.483024884" Oct 29 23:05:18.553283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:18.553198 5367 generic.go:356] "Generic (PLEG): container finished" podID="f15205b7-2c11-45b1-8f4d-431f4c38c8e4" containerID="5852389f96f1e0faf461a4615144e31b79d7e2d9227bbd1961bfa53809f05bdf" exitCode=0 Oct 29 23:05:18.553686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:18.553285 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" event={"ID":"f15205b7-2c11-45b1-8f4d-431f4c38c8e4","Type":"ContainerDied","Data":"5852389f96f1e0faf461a4615144e31b79d7e2d9227bbd1961bfa53809f05bdf"} Oct 29 23:05:18.555060 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:18.555034 5367 generic.go:356] "Generic (PLEG): container finished" podID="670ca2f2-0a4e-477c-a707-f727fbcf9dc6" containerID="94a8024f8ad997263ab790e1833f9af045568776b3dd972ce38cf9a638aa028c" exitCode=0 Oct 29 23:05:18.555197 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:18.555059 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" event={"ID":"670ca2f2-0a4e-477c-a707-f727fbcf9dc6","Type":"ContainerDied","Data":"94a8024f8ad997263ab790e1833f9af045568776b3dd972ce38cf9a638aa028c"} Oct 29 23:05:19.729414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.729387 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:19.764256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.764230 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:19.838461 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838379 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p\" (UniqueName: \"kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p\") pod \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " Oct 29 23:05:19.838649 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838490 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca\") pod \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " Oct 29 23:05:19.838649 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838550 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca\") pod \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " Oct 29 23:05:19.838649 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838619 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j\" (UniqueName: \"kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j\") pod \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " Oct 29 23:05:19.838872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838689 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca\") pod \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\" (UID: \"f15205b7-2c11-45b1-8f4d-431f4c38c8e4\") " Oct 29 23:05:19.838872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838771 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca\") pod \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\" (UID: \"670ca2f2-0a4e-477c-a707-f727fbcf9dc6\") " Oct 29 23:05:19.838974 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838909 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca" pod "670ca2f2-0a4e-477c-a707-f727fbcf9dc6" (UID: "670ca2f2-0a4e-477c-a707-f727fbcf9dc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:19.838974 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.838934 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca" pod "f15205b7-2c11-45b1-8f4d-431f4c38c8e4" (UID: "f15205b7-2c11-45b1-8f4d-431f4c38c8e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:19.839343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.839321 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:19.839343 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.839346 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:19.841211 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.841185 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p" pod "f15205b7-2c11-45b1-8f4d-431f4c38c8e4" (UID: "f15205b7-2c11-45b1-8f4d-431f4c38c8e4"). InnerVolumeSpecName "kube-api-access-lz87p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:19.841531 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.841510 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca" pod "f15205b7-2c11-45b1-8f4d-431f4c38c8e4" (UID: "f15205b7-2c11-45b1-8f4d-431f4c38c8e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:19.841771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.841742 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca" pod "670ca2f2-0a4e-477c-a707-f727fbcf9dc6" (UID: "670ca2f2-0a4e-477c-a707-f727fbcf9dc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:19.841883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.841788 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j" pod "670ca2f2-0a4e-477c-a707-f727fbcf9dc6" (UID: "670ca2f2-0a4e-477c-a707-f727fbcf9dc6"). InnerVolumeSpecName "kube-api-access-64w6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:19.940500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.940470 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz87p\" (UniqueName: \"kubernetes.io/projected/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-kube-api-access-lz87p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:19.940500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.940496 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64w6j\" (UniqueName: \"kubernetes.io/projected/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-kube-api-access-64w6j\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:19.940655 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.940506 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15205b7-2c11-45b1-8f4d-431f4c38c8e4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:19.940655 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:19.940516 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670ca2f2-0a4e-477c-a707-f727fbcf9dc6-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.573291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:20.573262 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" Oct 29 23:05:20.573452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:20.573302 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5" event={"ID":"670ca2f2-0a4e-477c-a707-f727fbcf9dc6","Type":"ContainerDied","Data":"0e1bddfaf2a9da832628d7c2de9bfe0f69e8206e66cce576cc59c21e29254c95"} Oct 29 23:05:20.573452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:20.573347 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1bddfaf2a9da832628d7c2de9bfe0f69e8206e66cce576cc59c21e29254c95" Oct 29 23:05:20.575099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:20.575077 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" event={"ID":"f15205b7-2c11-45b1-8f4d-431f4c38c8e4","Type":"ContainerDied","Data":"b99ce9d8605d1c0f0dd4290d8baad91b7f49a2b21ae75c55165e76dd89dbc59c"} Oct 29 23:05:20.575255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:20.575104 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf" Oct 29 23:05:20.575255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:20.575112 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b99ce9d8605d1c0f0dd4290d8baad91b7f49a2b21ae75c55165e76dd89dbc59c" Oct 29 23:05:21.378548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.378507 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" podUID="e52d37f9-e67b-43a3-849f-0b9f0077f13d" containerName="payload-processing" containerID="cri-o://02511be994abc0527f89cc533f1acb390d70c835e8c8365c045b46dea95c469e" gracePeriod=25 Oct 29 23:05:21.425373 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.425327 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" podUID="afd57bb9-a797-4434-be8a-4550badfdbeb" containerName="payload-pre-processing" containerID="cri-o://cad417c5cc294f2fe00ba19e6d530f815fd84fcda89aa756d1983dc7ef379fb3" gracePeriod=25 Oct 29 23:05:21.561795 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.561771 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 23:05:21.587498 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.587468 5367 generic.go:356] "Generic (PLEG): container finished" podID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerID="9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099" exitCode=0 Oct 29 23:05:21.587683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.587661 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" Oct 29 23:05:21.587824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.587808 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" event={"ID":"466bb494-cc30-4ab1-b12d-b6157e64f152","Type":"ContainerDied","Data":"9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099"} Oct 29 23:05:21.587866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.587844 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp" event={"ID":"466bb494-cc30-4ab1-b12d-b6157e64f152","Type":"ContainerDied","Data":"d76ee21bcfb29d02ef0abb594678cb3f22316a71356288d6f96b0a9fc3b02d05"} Oct 29 23:05:21.587907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.587867 5367 scope.go:122] "RemoveContainer" containerID="9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099" Oct 29 23:05:21.590245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.590217 5367 generic.go:356] "Generic (PLEG): container finished" podID="afd57bb9-a797-4434-be8a-4550badfdbeb" containerID="cad417c5cc294f2fe00ba19e6d530f815fd84fcda89aa756d1983dc7ef379fb3" exitCode=0 Oct 29 23:05:21.590390 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.590363 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" event={"ID":"afd57bb9-a797-4434-be8a-4550badfdbeb","Type":"ContainerDied","Data":"cad417c5cc294f2fe00ba19e6d530f815fd84fcda89aa756d1983dc7ef379fb3"} Oct 29 23:05:21.592235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.592207 5367 generic.go:356] "Generic (PLEG): container finished" podID="e52d37f9-e67b-43a3-849f-0b9f0077f13d" containerID="02511be994abc0527f89cc533f1acb390d70c835e8c8365c045b46dea95c469e" exitCode=0 Oct 29 23:05:21.592364 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.592339 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" event={"ID":"e52d37f9-e67b-43a3-849f-0b9f0077f13d","Type":"ContainerDied","Data":"02511be994abc0527f89cc533f1acb390d70c835e8c8365c045b46dea95c469e"} Oct 29 23:05:21.600738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.600614 5367 scope.go:122] "RemoveContainer" containerID="69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8" Oct 29 23:05:21.619955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.619928 5367 scope.go:122] "RemoveContainer" containerID="9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099" Oct 29 23:05:21.620532 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:21.620504 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099\": container with ID starting with 9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099 not found: ID does not exist" containerID="9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099" Oct 29 23:05:21.620647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.620540 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099"} err="failed to get container status \"9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099\": rpc error: code = NotFound desc = could not find container \"9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099\": container with ID starting with 9947030ca5bc74697af0a59f48f41cd749a54f1a78fe8f802b9b1df515d54099 not found: ID does not exist" Oct 29 23:05:21.620647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.620580 5367 scope.go:122] "RemoveContainer" containerID="69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8" Oct 29 23:05:21.622894 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:21.622787 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8\": container with ID starting with 69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8 not found: ID does not exist" containerID="69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8" Oct 29 23:05:21.622894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.622836 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8"} err="failed to get container status \"69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8\": rpc error: code = NotFound desc = could not find container \"69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8\": container with ID starting with 69431aebca8e772647bd9a235d3953fbdfcab5ce723602390a224534fa3bdbe8 not found: ID does not exist" Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.662462 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.662677 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.662777 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.662832 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.662883 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.662956 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.663064 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.663130 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.663752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.663231 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo\") pod \"466bb494-cc30-4ab1-b12d-b6157e64f152\" (UID: \"466bb494-cc30-4ab1-b12d-b6157e64f152\") " Oct 29 23:05:21.664522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.664059 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:21.664522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.664285 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:21.664522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.664332 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:21.664522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.664365 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:21.664771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.664616 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:21.667317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.667288 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:21.667317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.667305 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "kube-api-access-ghs58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:21.667544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.667523 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:21.670720 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.670676 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token" pod "466bb494-cc30-4ab1-b12d-b6157e64f152" (UID: "466bb494-cc30-4ab1-b12d-b6157e64f152"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:21.689059 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.689028 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 23:05:21.753372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.753349 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 23:05:21.765353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.765321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume\") pod \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " Oct 29 23:05:21.765545 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.765523 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4\" (UniqueName: \"kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4\") pod \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\" (UID: \"e52d37f9-e67b-43a3-849f-0b9f0077f13d\") " Oct 29 23:05:21.765642 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.765622 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume" pod "e52d37f9-e67b-43a3-849f-0b9f0077f13d" (UID: "e52d37f9-e67b-43a3-849f-0b9f0077f13d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:21.766185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766164 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766192 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghs58\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-kube-api-access-ghs58\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766207 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766226 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466bb494-cc30-4ab1-b12d-b6157e64f152-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766238 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766253 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766266 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466bb494-cc30-4ab1-b12d-b6157e64f152-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766277 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766290 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d37f9-e67b-43a3-849f-0b9f0077f13d-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.766423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.766304 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466bb494-cc30-4ab1-b12d-b6157e64f152-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.767636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.767608 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4" pod "e52d37f9-e67b-43a3-849f-0b9f0077f13d" (UID: "e52d37f9-e67b-43a3-849f-0b9f0077f13d"). InnerVolumeSpecName "kube-api-access-p94c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:21.867515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.867477 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume\") pod \"afd57bb9-a797-4434-be8a-4550badfdbeb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " Oct 29 23:05:21.867792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.867774 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t\" (UniqueName: \"kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t\") pod \"afd57bb9-a797-4434-be8a-4550badfdbeb\" (UID: \"afd57bb9-a797-4434-be8a-4550badfdbeb\") " Oct 29 23:05:21.867884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.867849 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume" pod "afd57bb9-a797-4434-be8a-4550badfdbeb" (UID: "afd57bb9-a797-4434-be8a-4550badfdbeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:21.868170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.868151 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p94c4\" (UniqueName: \"kubernetes.io/projected/e52d37f9-e67b-43a3-849f-0b9f0077f13d-kube-api-access-p94c4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.868231 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.868174 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd57bb9-a797-4434-be8a-4550badfdbeb-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:21.869834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.869813 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t" pod "afd57bb9-a797-4434-be8a-4550badfdbeb" (UID: "afd57bb9-a797-4434-be8a-4550badfdbeb"). InnerVolumeSpecName "kube-api-access-4gh9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:21.938458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.938427 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp"] Oct 29 23:05:21.941877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.941857 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0906f4-openshift-default-7b6bf8dd5f-4wncp"] Oct 29 23:05:21.969388 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:21.969365 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gh9t\" (UniqueName: \"kubernetes.io/projected/afd57bb9-a797-4434-be8a-4550badfdbeb-kube-api-access-4gh9t\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.331230 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:22.331191 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:05:22.604455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.604374 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" Oct 29 23:05:22.604455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.604401 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb" event={"ID":"afd57bb9-a797-4434-be8a-4550badfdbeb","Type":"ContainerDied","Data":"31c6c4e9165de629619ca275fe0c8cc588a78f2d7fd285525b37eb13bbd327c2"} Oct 29 23:05:22.605040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.604476 5367 scope.go:122] "RemoveContainer" containerID="cad417c5cc294f2fe00ba19e6d530f815fd84fcda89aa756d1983dc7ef379fb3" Oct 29 23:05:22.612175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.612156 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" Oct 29 23:05:22.612319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.612235 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d" event={"ID":"e52d37f9-e67b-43a3-849f-0b9f0077f13d","Type":"ContainerDied","Data":"e14a9a4d1fe2514b4009ef9a5586d90312b9e707b51dc0248dda08f08e4e8dfe"} Oct 29 23:05:22.618538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.618519 5367 scope.go:122] "RemoveContainer" containerID="02511be994abc0527f89cc533f1acb390d70c835e8c8365c045b46dea95c469e" Oct 29 23:05:22.678868 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.678834 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d"] Oct 29 23:05:22.688388 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.688356 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0d12c5-dc776f966-t282d"] Oct 29 23:05:22.724674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.724644 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb"] Oct 29 23:05:22.728076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.728055 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0d12c5-d9d745775-mvczb"] Oct 29 23:05:22.865004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.864919 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb"] Oct 29 23:05:22.874203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:22.874176 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ee1fc-9dlrb"] Oct 29 23:05:23.222546 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:23.222419 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09607296-b9c5-405a-b307-195224c0d1fb" path="/var/lib/kubelet/pods/09607296-b9c5-405a-b307-195224c0d1fb/volumes" Oct 29 23:05:23.223100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:23.223078 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" path="/var/lib/kubelet/pods/466bb494-cc30-4ab1-b12d-b6157e64f152/volumes" Oct 29 23:05:23.223587 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:23.223563 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd57bb9-a797-4434-be8a-4550badfdbeb" path="/var/lib/kubelet/pods/afd57bb9-a797-4434-be8a-4550badfdbeb/volumes" Oct 29 23:05:23.224324 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:23.224308 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52d37f9-e67b-43a3-849f-0b9f0077f13d" path="/var/lib/kubelet/pods/e52d37f9-e67b-43a3-849f-0b9f0077f13d/volumes" Oct 29 23:05:25.658355 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:25.658322 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" containerID="cri-o://499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7" gracePeriod=45 Oct 29 23:05:26.019918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.019892 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 23:05:26.124978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.124948 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir\") pod \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " Oct 29 23:05:26.125141 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125055 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache\") pod \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " Oct 29 23:05:26.125273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125139 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs\") pod \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " Oct 29 23:05:26.125273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125208 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm\") pod \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " Oct 29 23:05:26.125493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125307 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5\" (UniqueName: \"kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5\") pod \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " Oct 29 23:05:26.125493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125294 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir" pod "a7eea3c5-0fe1-42ce-876c-e44421cd3db1" (UID: "a7eea3c5-0fe1-42ce-876c-e44421cd3db1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.125493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125337 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache" pod "a7eea3c5-0fe1-42ce-876c-e44421cd3db1" (UID: "a7eea3c5-0fe1-42ce-876c-e44421cd3db1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.125493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.125402 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home\") pod \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\" (UID: \"a7eea3c5-0fe1-42ce-876c-e44421cd3db1\") " Oct 29 23:05:26.126277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.126022 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.126277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.126048 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.126649 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.126604 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home" pod "a7eea3c5-0fe1-42ce-876c-e44421cd3db1" (UID: "a7eea3c5-0fe1-42ce-876c-e44421cd3db1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.128149 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.128125 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5" pod "a7eea3c5-0fe1-42ce-876c-e44421cd3db1" (UID: "a7eea3c5-0fe1-42ce-876c-e44421cd3db1"). InnerVolumeSpecName "kube-api-access-rljp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:26.128239 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.128191 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs" pod "a7eea3c5-0fe1-42ce-876c-e44421cd3db1" (UID: "a7eea3c5-0fe1-42ce-876c-e44421cd3db1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:26.128239 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.128228 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm" pod "a7eea3c5-0fe1-42ce-876c-e44421cd3db1" (UID: "a7eea3c5-0fe1-42ce-876c-e44421cd3db1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.227497 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.227468 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.227497 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.227494 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.227497 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.227504 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljp5\" (UniqueName: \"kubernetes.io/projected/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-kube-api-access-rljp5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.227762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.227514 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eea3c5-0fe1-42ce-876c-e44421cd3db1-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.262599 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.262521 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk"] Oct 29 23:05:26.263252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263230 5367 cpu_manager.go:414] "containerMap: removing container" podUID="afd57bb9-a797-4434-be8a-4550badfdbeb" containerName="payload-pre-processing" Oct 29 23:05:26.263252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263251 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd57bb9-a797-4434-be8a-4550badfdbeb" containerName="payload-pre-processing" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263267 5367 cpu_manager.go:414] "containerMap: removing container" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263272 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263288 5367 cpu_manager.go:414] "containerMap: removing container" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263293 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263303 5367 cpu_manager.go:414] "containerMap: removing container" podUID="670ca2f2-0a4e-477c-a707-f727fbcf9dc6" containerName="revoke-keys" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263307 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670ca2f2-0a4e-477c-a707-f727fbcf9dc6" containerName="revoke-keys" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263321 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e52d37f9-e67b-43a3-849f-0b9f0077f13d" containerName="payload-processing" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263325 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52d37f9-e67b-43a3-849f-0b9f0077f13d" containerName="payload-processing" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263334 5367 cpu_manager.go:414] "containerMap: removing container" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263341 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263354 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f15205b7-2c11-45b1-8f4d-431f4c38c8e4" containerName="revoke-keys" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263359 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15205b7-2c11-45b1-8f4d-431f4c38c8e4" containerName="revoke-keys" Oct 29 23:05:26.263446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263453 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52d37f9-e67b-43a3-849f-0b9f0077f13d" containerName="payload-processing" Oct 29 23:05:26.263915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263461 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerName="main" Oct 29 23:05:26.263915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263470 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15205b7-2c11-45b1-8f4d-431f4c38c8e4" containerName="revoke-keys" Oct 29 23:05:26.263915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263479 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd57bb9-a797-4434-be8a-4550badfdbeb" containerName="payload-pre-processing" Oct 29 23:05:26.263915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263486 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" Oct 29 23:05:26.263915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263493 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="466bb494-cc30-4ab1-b12d-b6157e64f152" containerName="istio-proxy" Oct 29 23:05:26.263915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.263500 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="670ca2f2-0a4e-477c-a707-f727fbcf9dc6" containerName="revoke-keys" Oct 29 23:05:26.267186 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.267167 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.272516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.272491 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5"] Oct 29 23:05:26.276365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.276348 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.285091 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.285053 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:26.285210 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.285163 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:05:26.286215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.286193 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk"] Oct 29 23:05:26.297418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.297387 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5"] Oct 29 23:05:26.430280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.430250 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.430490 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.430301 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnq2\" (UniqueName: \"kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.430490 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.430412 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.430490 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.430454 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.430615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.430543 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.430615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.430585 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48glq\" (UniqueName: \"kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.531905 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.531814 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.531905 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.531864 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.531905 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.531901 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.532187 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.531943 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48glq\" (UniqueName: \"kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.532187 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.532067 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.532187 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.532115 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnq2\" (UniqueName: \"kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.533118 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.533091 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.533266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.533244 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.534875 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.534849 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.534875 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.534865 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.553841 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.553818 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48glq\" (UniqueName: \"kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq\") pod \"maas-api-revoke-sub-bb61c20a-sxxt5\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.553945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.553848 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnq2\" (UniqueName: \"kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2\") pod \"maas-api-revoke-sub-901d3578-zkcrk\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.581513 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.581493 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:26.593250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.593229 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:26.711116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.710329 5367 generic.go:356] "Generic (PLEG): container finished" podID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" containerID="499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7" exitCode=0 Oct 29 23:05:26.711116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.710374 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" Oct 29 23:05:26.711116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.710459 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" event={"ID":"a7eea3c5-0fe1-42ce-876c-e44421cd3db1","Type":"ContainerDied","Data":"499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7"} Oct 29 23:05:26.711116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.710528 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7" event={"ID":"a7eea3c5-0fe1-42ce-876c-e44421cd3db1","Type":"ContainerDied","Data":"79a2e13a323aa1162bb6e094d8c2e63854c5b77c0344c915aeeba7e0bc7c4d26"} Oct 29 23:05:26.711116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.710554 5367 scope.go:122] "RemoveContainer" containerID="499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7" Oct 29 23:05:26.733592 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.733569 5367 scope.go:122] "RemoveContainer" containerID="499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7" Oct 29 23:05:26.733963 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:26.733925 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7\": container with ID starting with 499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7 not found: ID does not exist" containerID="499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7" Oct 29 23:05:26.734052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.733964 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7"} err="failed to get container status \"499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7\": rpc error: code = NotFound desc = could not find container \"499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7\": container with ID starting with 499f56afe7b2eec342250c1fb3d14e9430b21f254072e99487d8edfb91f528a7 not found: ID does not exist" Oct 29 23:05:26.775749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.774142 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7"] Oct 29 23:05:26.778832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.778802 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-2-w5-0d12c5-kserve-76d7c59cbd-9bck7"] Oct 29 23:05:26.920340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.920311 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5"] Oct 29 23:05:26.924300 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:26.924274 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852c31db_fafa_4c58_8b6e_c89034730fc6.slice/crio-f2f61cd26e31f698421a394f5adf80391c8ab6a5cf216f991e26500e61e9f35f WatchSource:0}: Error finding container f2f61cd26e31f698421a394f5adf80391c8ab6a5cf216f991e26500e61e9f35f: Status 404 returned error can't find the container with id f2f61cd26e31f698421a394f5adf80391c8ab6a5cf216f991e26500e61e9f35f Oct 29 23:05:26.960896 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:26.960291 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk"] Oct 29 23:05:27.222379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.222287 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7eea3c5-0fe1-42ce-876c-e44421cd3db1" path="/var/lib/kubelet/pods/a7eea3c5-0fe1-42ce-876c-e44421cd3db1/volumes" Oct 29 23:05:27.720361 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.720326 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" event={"ID":"852c31db-fafa-4c58-8b6e-c89034730fc6","Type":"ContainerStarted","Data":"67c8bd646e3e3c7b5b275aba4b0a060214aea22058f5e2b1ae9b5bce32d62f2d"} Oct 29 23:05:27.720849 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.720387 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" event={"ID":"852c31db-fafa-4c58-8b6e-c89034730fc6","Type":"ContainerStarted","Data":"f2f61cd26e31f698421a394f5adf80391c8ab6a5cf216f991e26500e61e9f35f"} Oct 29 23:05:27.721997 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.721968 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" event={"ID":"c3c4e342-2995-49f9-b478-4d93a4a012d4","Type":"ContainerStarted","Data":"2740a6eca586da1f0110dffdd93261203b98b482f60fb56b89eca5184abf0d55"} Oct 29 23:05:27.722123 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.722015 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" event={"ID":"c3c4e342-2995-49f9-b478-4d93a4a012d4","Type":"ContainerStarted","Data":"0139a866652bd8962df1e93062dbe8d86b0eadac68eb1b3c5a28de9b9aabe3b3"} Oct 29 23:05:27.765802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.765757 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" podStartSLOduration=1.7657451389999999 podStartE2EDuration="1.765745139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:27.763888321 +0000 UTC m=+2994.657963242" watchObservedRunningTime="2026-10-29 23:05:27.765745139 +0000 UTC m=+2994.659820059" Oct 29 23:05:27.824368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:27.824303 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" podStartSLOduration=1.824286091 podStartE2EDuration="1.824286091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:27.807020051 +0000 UTC m=+2994.701094974" watchObservedRunningTime="2026-10-29 23:05:27.824286091 +0000 UTC m=+2994.718361011" Oct 29 23:05:28.360120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.360085 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b"] Oct 29 23:05:28.360316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.360292 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" podUID="c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" containerName="payload-processing" containerID="cri-o://cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253" gracePeriod=30 Oct 29 23:05:28.639273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.639240 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:05:28.679607 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.679568 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume\") pod \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " Oct 29 23:05:28.679934 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.679909 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh\" (UniqueName: \"kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh\") pod \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\" (UID: \"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f\") " Oct 29 23:05:28.680154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.680133 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume" pod "c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" (UID: "c541f86d-7b0a-436c-a2eb-e6c4452e2a0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:28.680432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.680406 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.682047 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.682030 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh" pod "c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" (UID: "c541f86d-7b0a-436c-a2eb-e6c4452e2a0f"). InnerVolumeSpecName "kube-api-access-9tkmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:28.733306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.733270 5367 generic.go:356] "Generic (PLEG): container finished" podID="c3c4e342-2995-49f9-b478-4d93a4a012d4" containerID="2740a6eca586da1f0110dffdd93261203b98b482f60fb56b89eca5184abf0d55" exitCode=0 Oct 29 23:05:28.733737 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.733361 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" event={"ID":"c3c4e342-2995-49f9-b478-4d93a4a012d4","Type":"ContainerDied","Data":"2740a6eca586da1f0110dffdd93261203b98b482f60fb56b89eca5184abf0d55"} Oct 29 23:05:28.735244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.735218 5367 generic.go:356] "Generic (PLEG): container finished" podID="c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" containerID="cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253" exitCode=0 Oct 29 23:05:28.735389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.735257 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" Oct 29 23:05:28.735389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.735309 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" event={"ID":"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f","Type":"ContainerDied","Data":"cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253"} Oct 29 23:05:28.735389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.735369 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b" event={"ID":"c541f86d-7b0a-436c-a2eb-e6c4452e2a0f","Type":"ContainerDied","Data":"e5d3631ef4b5702bd5046a0d16eff9f26d2775ddcf4084314480ee0f171e409e"} Oct 29 23:05:28.735561 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.735400 5367 scope.go:122] "RemoveContainer" containerID="cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253" Oct 29 23:05:28.739066 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.739043 5367 generic.go:356] "Generic (PLEG): container finished" podID="852c31db-fafa-4c58-8b6e-c89034730fc6" containerID="67c8bd646e3e3c7b5b275aba4b0a060214aea22058f5e2b1ae9b5bce32d62f2d" exitCode=0 Oct 29 23:05:28.739194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.739128 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" event={"ID":"852c31db-fafa-4c58-8b6e-c89034730fc6","Type":"ContainerDied","Data":"67c8bd646e3e3c7b5b275aba4b0a060214aea22058f5e2b1ae9b5bce32d62f2d"} Oct 29 23:05:28.748984 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.748964 5367 scope.go:122] "RemoveContainer" containerID="cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253" Oct 29 23:05:28.749338 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:28.749311 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253\": container with ID starting with cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253 not found: ID does not exist" containerID="cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253" Oct 29 23:05:28.749535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.749350 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253"} err="failed to get container status \"cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253\": rpc error: code = NotFound desc = could not find container \"cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253\": container with ID starting with cd2ef25d51c8d2850fc13ad03679fa9c88edc616341f55c35d0da65f874ad253 not found: ID does not exist" Oct 29 23:05:28.799735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.794573 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tkmh\" (UniqueName: \"kubernetes.io/projected/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f-kube-api-access-9tkmh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.882227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.882146 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b"] Oct 29 23:05:28.885529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.885503 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3e2b2c-bb56cc65b-5fg8b"] Oct 29 23:05:28.966800 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.966772 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5"] Oct 29 23:05:28.968188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:28.968168 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3e2b2c-wg6q5"] Oct 29 23:05:29.223127 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:29.223050 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670ca2f2-0a4e-477c-a707-f727fbcf9dc6" path="/var/lib/kubelet/pods/670ca2f2-0a4e-477c-a707-f727fbcf9dc6/volumes" Oct 29 23:05:29.223625 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:29.223609 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" path="/var/lib/kubelet/pods/c541f86d-7b0a-436c-a2eb-e6c4452e2a0f/volumes" Oct 29 23:05:29.900177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:29.900150 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:29.937280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:29.937199 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:30.011808 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.011723 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq\" (UniqueName: \"kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq\") pod \"852c31db-fafa-4c58-8b6e-c89034730fc6\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " Oct 29 23:05:30.011996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.011935 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca\") pod \"852c31db-fafa-4c58-8b6e-c89034730fc6\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " Oct 29 23:05:30.012071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.012047 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca\") pod \"c3c4e342-2995-49f9-b478-4d93a4a012d4\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " Oct 29 23:05:30.012136 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.012118 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca\") pod \"c3c4e342-2995-49f9-b478-4d93a4a012d4\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " Oct 29 23:05:30.012208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.012157 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca\") pod \"852c31db-fafa-4c58-8b6e-c89034730fc6\" (UID: \"852c31db-fafa-4c58-8b6e-c89034730fc6\") " Oct 29 23:05:30.012208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.012188 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2\" (UniqueName: \"kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2\") pod \"c3c4e342-2995-49f9-b478-4d93a4a012d4\" (UID: \"c3c4e342-2995-49f9-b478-4d93a4a012d4\") " Oct 29 23:05:30.012736 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.012682 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca" pod "c3c4e342-2995-49f9-b478-4d93a4a012d4" (UID: "c3c4e342-2995-49f9-b478-4d93a4a012d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:30.012918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.012743 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca" pod "852c31db-fafa-4c58-8b6e-c89034730fc6" (UID: "852c31db-fafa-4c58-8b6e-c89034730fc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:30.014363 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.014326 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq" pod "852c31db-fafa-4c58-8b6e-c89034730fc6" (UID: "852c31db-fafa-4c58-8b6e-c89034730fc6"). InnerVolumeSpecName "kube-api-access-48glq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:30.014833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.014795 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2" pod "c3c4e342-2995-49f9-b478-4d93a4a012d4" (UID: "c3c4e342-2995-49f9-b478-4d93a4a012d4"). InnerVolumeSpecName "kube-api-access-jjnq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:30.015143 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.015117 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca" pod "852c31db-fafa-4c58-8b6e-c89034730fc6" (UID: "852c31db-fafa-4c58-8b6e-c89034730fc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:30.015256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.015235 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca" pod "c3c4e342-2995-49f9-b478-4d93a4a012d4" (UID: "c3c4e342-2995-49f9-b478-4d93a4a012d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:30.114082 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.114048 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:30.114082 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.114076 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:30.114280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.114089 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4e342-2995-49f9-b478-4d93a4a012d4-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:30.114280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.114103 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852c31db-fafa-4c58-8b6e-c89034730fc6-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:30.114280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.114114 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnq2\" (UniqueName: \"kubernetes.io/projected/c3c4e342-2995-49f9-b478-4d93a4a012d4-kube-api-access-jjnq2\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:30.114280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.114128 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48glq\" (UniqueName: \"kubernetes.io/projected/852c31db-fafa-4c58-8b6e-c89034730fc6-kube-api-access-48glq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:30.795732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.792386 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" event={"ID":"852c31db-fafa-4c58-8b6e-c89034730fc6","Type":"ContainerDied","Data":"f2f61cd26e31f698421a394f5adf80391c8ab6a5cf216f991e26500e61e9f35f"} Oct 29 23:05:30.795732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.792461 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2f61cd26e31f698421a394f5adf80391c8ab6a5cf216f991e26500e61e9f35f" Oct 29 23:05:30.795732 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.792539 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5" Oct 29 23:05:30.799719 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.798277 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" event={"ID":"c3c4e342-2995-49f9-b478-4d93a4a012d4","Type":"ContainerDied","Data":"0139a866652bd8962df1e93062dbe8d86b0eadac68eb1b3c5a28de9b9aabe3b3"} Oct 29 23:05:30.799719 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.798334 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0139a866652bd8962df1e93062dbe8d86b0eadac68eb1b3c5a28de9b9aabe3b3" Oct 29 23:05:30.799719 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:30.798401 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk" Oct 29 23:05:31.077573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.077535 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j"] Oct 29 23:05:31.078289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078267 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" containerName="payload-processing" Oct 29 23:05:31.078394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078289 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" containerName="payload-processing" Oct 29 23:05:31.078394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078322 5367 cpu_manager.go:414] "containerMap: removing container" podUID="852c31db-fafa-4c58-8b6e-c89034730fc6" containerName="revoke-keys" Oct 29 23:05:31.078394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078331 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852c31db-fafa-4c58-8b6e-c89034730fc6" containerName="revoke-keys" Oct 29 23:05:31.078394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078353 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c3c4e342-2995-49f9-b478-4d93a4a012d4" containerName="revoke-keys" Oct 29 23:05:31.078394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078361 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c4e342-2995-49f9-b478-4d93a4a012d4" containerName="revoke-keys" Oct 29 23:05:31.078689 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078562 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c541f86d-7b0a-436c-a2eb-e6c4452e2a0f" containerName="payload-processing" Oct 29 23:05:31.078689 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078579 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="852c31db-fafa-4c58-8b6e-c89034730fc6" containerName="revoke-keys" Oct 29 23:05:31.078689 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.078591 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c4e342-2995-49f9-b478-4d93a4a012d4" containerName="revoke-keys" Oct 29 23:05:31.082500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.082461 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.098788 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.098685 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:31.099023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.099003 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:05:31.102335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.102316 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j"] Oct 29 23:05:31.152979 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.152952 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.153134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.152986 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7fj\" (UniqueName: \"kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.153134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.153111 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.254781 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.254748 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.254952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.254786 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7fj\" (UniqueName: \"kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.254952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.254848 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.255486 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.255456 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.257120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.257098 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.276359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.276338 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7fj\" (UniqueName: \"kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj\") pod \"maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.395739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.395622 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:31.696048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.696022 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j"] Oct 29 23:05:31.696506 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:31.696479 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa6b708_452b_462b_a930_d6146f7201d8.slice/crio-e277fd61e017f5e7551053bf6818177bd2f63c70d44db49910fed5918cf30581 WatchSource:0}: Error finding container e277fd61e017f5e7551053bf6818177bd2f63c70d44db49910fed5918cf30581: Status 404 returned error can't find the container with id e277fd61e017f5e7551053bf6818177bd2f63c70d44db49910fed5918cf30581 Oct 29 23:05:31.807634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:31.807598 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" event={"ID":"4fa6b708-452b-462b-a930-d6146f7201d8","Type":"ContainerStarted","Data":"e277fd61e017f5e7551053bf6818177bd2f63c70d44db49910fed5918cf30581"} Oct 29 23:05:32.816945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:32.816912 5367 generic.go:356] "Generic (PLEG): container finished" podID="4fa6b708-452b-462b-a930-d6146f7201d8" containerID="e3cd8ace1628e33c4bd25ebe569383a8be3a7018331ff6304dbf3e25c550e457" exitCode=0 Oct 29 23:05:32.817344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:32.817000 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" event={"ID":"4fa6b708-452b-462b-a930-d6146f7201d8","Type":"ContainerDied","Data":"e3cd8ace1628e33c4bd25ebe569383a8be3a7018331ff6304dbf3e25c550e457"} Oct 29 23:05:33.987012 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:33.986989 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:34.090088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.089999 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj\" (UniqueName: \"kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj\") pod \"4fa6b708-452b-462b-a930-d6146f7201d8\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " Oct 29 23:05:34.090267 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.090139 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca\") pod \"4fa6b708-452b-462b-a930-d6146f7201d8\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " Oct 29 23:05:34.090267 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.090247 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca\") pod \"4fa6b708-452b-462b-a930-d6146f7201d8\" (UID: \"4fa6b708-452b-462b-a930-d6146f7201d8\") " Oct 29 23:05:34.090663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.090634 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca" pod "4fa6b708-452b-462b-a930-d6146f7201d8" (UID: "4fa6b708-452b-462b-a930-d6146f7201d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:34.092168 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.092139 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj" pod "4fa6b708-452b-462b-a930-d6146f7201d8" (UID: "4fa6b708-452b-462b-a930-d6146f7201d8"). InnerVolumeSpecName "kube-api-access-kk7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:34.092288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.092244 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca" pod "4fa6b708-452b-462b-a930-d6146f7201d8" (UID: "4fa6b708-452b-462b-a930-d6146f7201d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:34.191941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.191911 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7fj\" (UniqueName: \"kubernetes.io/projected/4fa6b708-452b-462b-a930-d6146f7201d8-kube-api-access-kk7fj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:34.191941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.191939 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:34.191941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.191948 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa6b708-452b-462b-a930-d6146f7201d8-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:34.836285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.836254 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" Oct 29 23:05:34.836461 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.836301 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j" event={"ID":"4fa6b708-452b-462b-a930-d6146f7201d8","Type":"ContainerDied","Data":"e277fd61e017f5e7551053bf6818177bd2f63c70d44db49910fed5918cf30581"} Oct 29 23:05:34.836461 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:34.836347 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e277fd61e017f5e7551053bf6818177bd2f63c70d44db49910fed5918cf30581" Oct 29 23:05:39.307968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:39.307935 5367 scope.go:122] "RemoveContainer" containerID="5083dfe177f5b000d8554131dead5580149e76b643451b7221f61d8a5bf5cdab" Oct 29 23:05:39.319075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:39.319052 5367 scope.go:122] "RemoveContainer" containerID="be0e806d0f9002e90f9d0229d02119303ae75b69d3c24054afef37965dac7152" Oct 29 23:05:39.328572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:39.328547 5367 scope.go:122] "RemoveContainer" containerID="977bbe9bf7ec6527fb97dddaae11869204c0f91322eb07147a8d84c6e698158a" Oct 29 23:05:39.370304 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:39.369876 5367 scope.go:122] "RemoveContainer" containerID="76c303c76ad842c38f7ca1ed060220693bf67078c58aa9dc72f4d08d31a1a95e" Oct 29 23:05:41.114389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.114355 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6"] Oct 29 23:05:41.114951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.114935 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4fa6b708-452b-462b-a930-d6146f7201d8" containerName="revoke-keys" Oct 29 23:05:41.115005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.114952 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa6b708-452b-462b-a930-d6146f7201d8" containerName="revoke-keys" Oct 29 23:05:41.115079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.115070 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa6b708-452b-462b-a930-d6146f7201d8" containerName="revoke-keys" Oct 29 23:05:41.119213 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.119188 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.135539 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.135515 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:41.137968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.137948 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6"] Oct 29 23:05:41.147336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.147310 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:05:41.204192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.204167 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.204293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.204201 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.204347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.204320 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9q8h\" (UniqueName: \"kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.306001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.305967 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.306174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.306016 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.306174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.306101 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9q8h\" (UniqueName: \"kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.306764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.306735 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.308529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.308506 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.328415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.328394 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9q8h\" (UniqueName: \"kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h\") pod \"maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.431946 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.431877 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:41.736892 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.736862 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6"] Oct 29 23:05:41.737378 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:41.737353 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758ed51f_570b_4598_bae8_5492736024be.slice/crio-6602a7e2723e2a1d73c59623486f05973f33e643f188183366c18e333de8f538 WatchSource:0}: Error finding container 6602a7e2723e2a1d73c59623486f05973f33e643f188183366c18e333de8f538: Status 404 returned error can't find the container with id 6602a7e2723e2a1d73c59623486f05973f33e643f188183366c18e333de8f538 Oct 29 23:05:41.890926 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:41.890889 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" event={"ID":"758ed51f-570b-4598-bae8-5492736024be","Type":"ContainerStarted","Data":"6602a7e2723e2a1d73c59623486f05973f33e643f188183366c18e333de8f538"} Oct 29 23:05:42.910581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:42.910503 5367 generic.go:356] "Generic (PLEG): container finished" podID="758ed51f-570b-4598-bae8-5492736024be" containerID="4b57a4a2727146cd6fa9fa949b03067fbe7b0806972080627042cd72fbfa4f92" exitCode=0 Oct 29 23:05:42.910987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:42.910601 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" event={"ID":"758ed51f-570b-4598-bae8-5492736024be","Type":"ContainerDied","Data":"4b57a4a2727146cd6fa9fa949b03067fbe7b0806972080627042cd72fbfa4f92"} Oct 29 23:05:44.091367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.091343 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:44.244523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.244443 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca\") pod \"758ed51f-570b-4598-bae8-5492736024be\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " Oct 29 23:05:44.244523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.244498 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca\") pod \"758ed51f-570b-4598-bae8-5492736024be\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " Oct 29 23:05:44.244780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.244578 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h\" (UniqueName: \"kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h\") pod \"758ed51f-570b-4598-bae8-5492736024be\" (UID: \"758ed51f-570b-4598-bae8-5492736024be\") " Oct 29 23:05:44.244853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.244825 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca" pod "758ed51f-570b-4598-bae8-5492736024be" (UID: "758ed51f-570b-4598-bae8-5492736024be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:44.245178 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.245160 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758ed51f-570b-4598-bae8-5492736024be-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:44.246584 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.246560 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca" pod "758ed51f-570b-4598-bae8-5492736024be" (UID: "758ed51f-570b-4598-bae8-5492736024be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:44.246807 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.246783 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h" pod "758ed51f-570b-4598-bae8-5492736024be" (UID: "758ed51f-570b-4598-bae8-5492736024be"). InnerVolumeSpecName "kube-api-access-m9q8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:44.346756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.346711 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9q8h\" (UniqueName: \"kubernetes.io/projected/758ed51f-570b-4598-bae8-5492736024be-kube-api-access-m9q8h\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:44.346756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.346746 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758ed51f-570b-4598-bae8-5492736024be-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:44.944749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.944685 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" Oct 29 23:05:44.944749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.944738 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6" event={"ID":"758ed51f-570b-4598-bae8-5492736024be","Type":"ContainerDied","Data":"6602a7e2723e2a1d73c59623486f05973f33e643f188183366c18e333de8f538"} Oct 29 23:05:44.944965 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:44.944786 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6602a7e2723e2a1d73c59623486f05973f33e643f188183366c18e333de8f538" Oct 29 23:05:45.629205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:45.629172 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k"] Oct 29 23:05:45.642262 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:45.642237 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0d12c5-86f8k"] Oct 29 23:05:47.222985 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:47.222951 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71c5ec8-8dff-44ca-9ec1-c98d0d34678f" path="/var/lib/kubelet/pods/d71c5ec8-8dff-44ca-9ec1-c98d0d34678f/volumes" Oct 29 23:05:47.509542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:47.509467 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf"] Oct 29 23:05:47.522668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:47.522640 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e81b56-fjfzf"] Oct 29 23:05:49.246492 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:49.246437 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15205b7-2c11-45b1-8f4d-431f4c38c8e4" path="/var/lib/kubelet/pods/f15205b7-2c11-45b1-8f4d-431f4c38c8e4/volumes" Oct 29 23:05:52.355193 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:52.355153 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:05:52.564999 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:52.564967 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc"] Oct 29 23:05:52.565202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:52.565179 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" containerID="cri-o://a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6" gracePeriod=30 Oct 29 23:05:53.531351 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.531312 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn"] Oct 29 23:05:53.531886 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.531567 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" podUID="8a50fe23-eb39-45a2-ba51-8db99c581762" containerName="maas-api" containerID="cri-o://a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41" gracePeriod=30 Oct 29 23:05:53.844634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.844609 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:05:53.897115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.897088 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8\" (UniqueName: \"kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8\") pod \"8a50fe23-eb39-45a2-ba51-8db99c581762\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " Oct 29 23:05:53.897353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.897140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls\") pod \"8a50fe23-eb39-45a2-ba51-8db99c581762\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " Oct 29 23:05:53.897353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.897174 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca\") pod \"8a50fe23-eb39-45a2-ba51-8db99c581762\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " Oct 29 23:05:53.897353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.897211 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs\") pod \"8a50fe23-eb39-45a2-ba51-8db99c581762\" (UID: \"8a50fe23-eb39-45a2-ba51-8db99c581762\") " Oct 29 23:05:53.898487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.898426 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca" pod "8a50fe23-eb39-45a2-ba51-8db99c581762" (UID: "8a50fe23-eb39-45a2-ba51-8db99c581762"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:53.900389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.900346 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs" pod "8a50fe23-eb39-45a2-ba51-8db99c581762" (UID: "8a50fe23-eb39-45a2-ba51-8db99c581762"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:53.900708 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.900669 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8" pod "8a50fe23-eb39-45a2-ba51-8db99c581762" (UID: "8a50fe23-eb39-45a2-ba51-8db99c581762"). InnerVolumeSpecName "kube-api-access-2vqj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:53.901203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.901171 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls" pod "8a50fe23-eb39-45a2-ba51-8db99c581762" (UID: "8a50fe23-eb39-45a2-ba51-8db99c581762"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:53.999480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.999399 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqj8\" (UniqueName: \"kubernetes.io/projected/8a50fe23-eb39-45a2-ba51-8db99c581762-kube-api-access-2vqj8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:53.999480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.999434 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:53.999480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.999449 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a50fe23-eb39-45a2-ba51-8db99c581762-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:53.999480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:53.999462 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a50fe23-eb39-45a2-ba51-8db99c581762-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:54.078806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.078716 5367 generic.go:356] "Generic (PLEG): container finished" podID="8a50fe23-eb39-45a2-ba51-8db99c581762" containerID="a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41" exitCode=0 Oct 29 23:05:54.079372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.079117 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" Oct 29 23:05:54.082570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.079139 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" event={"ID":"8a50fe23-eb39-45a2-ba51-8db99c581762","Type":"ContainerDied","Data":"a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41"} Oct 29 23:05:54.082570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.081991 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn" event={"ID":"8a50fe23-eb39-45a2-ba51-8db99c581762","Type":"ContainerDied","Data":"b4204d23e95ded17541e8ed622bbb4ea21ae4eac27dd8e89640e6f9de730bc3c"} Oct 29 23:05:54.082570 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.082034 5367 scope.go:122] "RemoveContainer" containerID="a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41" Oct 29 23:05:54.138756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.138735 5367 scope.go:122] "RemoveContainer" containerID="a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41" Oct 29 23:05:54.139113 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:54.139091 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41\": container with ID starting with a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41 not found: ID does not exist" containerID="a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41" Oct 29 23:05:54.139188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.139125 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41"} err="failed to get container status \"a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41\": rpc error: code = NotFound desc = could not find container \"a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41\": container with ID starting with a6f96e6351e87329a82290d3b422b801378758b5acba2f4c09191790f0eb8a41 not found: ID does not exist" Oct 29 23:05:54.164088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.164059 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn"] Oct 29 23:05:54.170912 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.170887 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eddb03-7577f488d8-f2ghn"] Oct 29 23:05:54.189121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.189095 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j"] Oct 29 23:05:54.192136 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:54.192116 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eddb03-5rm6j"] Oct 29 23:05:55.239782 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.239744 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa6b708-452b-462b-a930-d6146f7201d8" path="/var/lib/kubelet/pods/4fa6b708-452b-462b-a930-d6146f7201d8/volumes" Oct 29 23:05:55.240459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.240430 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a50fe23-eb39-45a2-ba51-8db99c581762" path="/var/lib/kubelet/pods/8a50fe23-eb39-45a2-ba51-8db99c581762/volumes" Oct 29 23:05:55.393489 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.393452 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv"] Oct 29 23:05:55.393778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.393744 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" containerID="cri-o://13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6" gracePeriod=30 Oct 29 23:05:55.977356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977326 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs"] Oct 29 23:05:55.977881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977867 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8a50fe23-eb39-45a2-ba51-8db99c581762" containerName="maas-api" Oct 29 23:05:55.977954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977883 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a50fe23-eb39-45a2-ba51-8db99c581762" containerName="maas-api" Oct 29 23:05:55.977954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977906 5367 cpu_manager.go:414] "containerMap: removing container" podUID="758ed51f-570b-4598-bae8-5492736024be" containerName="revoke-keys" Oct 29 23:05:55.977954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977910 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758ed51f-570b-4598-bae8-5492736024be" containerName="revoke-keys" Oct 29 23:05:55.978048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977983 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a50fe23-eb39-45a2-ba51-8db99c581762" containerName="maas-api" Oct 29 23:05:55.978048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.977995 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="758ed51f-570b-4598-bae8-5492736024be" containerName="revoke-keys" Oct 29 23:05:55.981285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:55.981269 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.000307 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.000285 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:56.000307 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.000299 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:05:56.004894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.004870 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs"] Oct 29 23:05:56.013751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.013725 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:05:56.013751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.013736 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:05:56.069720 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.069353 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkwj\" (UniqueName: \"kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.069720 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.069575 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.069720 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.069679 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.172088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.172054 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkwj\" (UniqueName: \"kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.172263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.172134 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.172263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.172185 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.172833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.172806 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.174556 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.174530 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.196775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.196748 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkwj\" (UniqueName: \"kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.294246 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.294184 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:56.599329 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:56.599300 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs"] Oct 29 23:05:56.600415 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:05:56.600381 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9b164_4ce6_4cac_8216_233a564ab6cd.slice/crio-2820f6e8e097fb9f4cbba81ae0e02306b1fbd6b8b0ee88e37d272e6c70505ff6 WatchSource:0}: Error finding container 2820f6e8e097fb9f4cbba81ae0e02306b1fbd6b8b0ee88e37d272e6c70505ff6: Status 404 returned error can't find the container with id 2820f6e8e097fb9f4cbba81ae0e02306b1fbd6b8b0ee88e37d272e6c70505ff6 Oct 29 23:05:57.106119 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.106070 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" event={"ID":"b6d9b164-4ce6-4cac-8216-233a564ab6cd","Type":"ContainerStarted","Data":"7fb755eb5fbc244e5068de33584f892c2c240163344ce7b44e9db2022305d65e"} Oct 29 23:05:57.106297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.106191 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" event={"ID":"b6d9b164-4ce6-4cac-8216-233a564ab6cd","Type":"ContainerStarted","Data":"2820f6e8e097fb9f4cbba81ae0e02306b1fbd6b8b0ee88e37d272e6c70505ff6"} Oct 29 23:05:57.152228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.152125 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" podStartSLOduration=2.152113528 podStartE2EDuration="2.152113528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:57.149079318 +0000 UTC m=+3024.043154239" watchObservedRunningTime="2026-10-29 23:05:57.152113528 +0000 UTC m=+3024.046188447" Oct 29 23:05:57.505222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.505151 5367 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:05:57.505222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.505208 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:05:57.699681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.699627 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:05:57.699890 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.699803 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:05:57.845957 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.845931 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 23:05:57.994361 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994327 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994433 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994501 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994544 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994597 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994640 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994664 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994786 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.994854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994832 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs\") pod \"44780614-e050-4e18-885d-52a560c5deb7\" (UID: \"44780614-e050-4e18-885d-52a560c5deb7\") " Oct 29 23:05:57.995028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994945 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.995028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.994961 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.995142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995090 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.995316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995240 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.995316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995295 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:57.995583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995563 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:57.995641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995591 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:57.995641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995606 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:57.995641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995618 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44780614-e050-4e18-885d-52a560c5deb7-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:57.995641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.995630 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:57.996756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.996730 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:57.996756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.996747 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.997023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.997002 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "kube-api-access-hgp77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:57.997128 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:57.997114 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo" pod "44780614-e050-4e18-885d-52a560c5deb7" (UID: "44780614-e050-4e18-885d-52a560c5deb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:58.096914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.096873 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.096914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.096909 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgp77\" (UniqueName: \"kubernetes.io/projected/44780614-e050-4e18-885d-52a560c5deb7-kube-api-access-hgp77\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.096914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.096921 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44780614-e050-4e18-885d-52a560c5deb7-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.096914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.096929 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44780614-e050-4e18-885d-52a560c5deb7-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.115131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.115100 5367 generic.go:356] "Generic (PLEG): container finished" podID="44780614-e050-4e18-885d-52a560c5deb7" containerID="a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6" exitCode=0 Oct 29 23:05:58.115249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.115153 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" Oct 29 23:05:58.115249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.115184 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" event={"ID":"44780614-e050-4e18-885d-52a560c5deb7","Type":"ContainerDied","Data":"a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6"} Oct 29 23:05:58.115249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.115240 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc" event={"ID":"44780614-e050-4e18-885d-52a560c5deb7","Type":"ContainerDied","Data":"19c61ae6544c18cdff76c2fee9e8125f1fdfd0cebba5747f4e2c9d0f764aba70"} Oct 29 23:05:58.115374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.115263 5367 scope.go:122] "RemoveContainer" containerID="a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6" Oct 29 23:05:58.117094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.117074 5367 generic.go:356] "Generic (PLEG): container finished" podID="b6d9b164-4ce6-4cac-8216-233a564ab6cd" containerID="7fb755eb5fbc244e5068de33584f892c2c240163344ce7b44e9db2022305d65e" exitCode=0 Oct 29 23:05:58.117216 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.117152 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" event={"ID":"b6d9b164-4ce6-4cac-8216-233a564ab6cd","Type":"ContainerDied","Data":"7fb755eb5fbc244e5068de33584f892c2c240163344ce7b44e9db2022305d65e"} Oct 29 23:05:58.126072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.126056 5367 scope.go:122] "RemoveContainer" containerID="a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6" Oct 29 23:05:58.126331 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:05:58.126313 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6\": container with ID starting with a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6 not found: ID does not exist" containerID="a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6" Oct 29 23:05:58.126377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.126340 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6"} err="failed to get container status \"a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6\": rpc error: code = NotFound desc = could not find container \"a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6\": container with ID starting with a69a50d49fcfc5ed504c8f3b4890fdaaea72745b1ef2890820979ee29ae0ccb6 not found: ID does not exist" Oct 29 23:05:58.219261 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.219233 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc"] Oct 29 23:05:58.223194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:58.223168 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-fpfrc"] Oct 29 23:05:59.053767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.053729 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5"] Oct 29 23:05:59.054281 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.053989 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" podUID="612a9ec7-d897-49a0-b224-aeae8495961d" containerName="payload-pre-processing" containerID="cri-o://26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7" gracePeriod=30 Oct 29 23:05:59.223551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.223518 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44780614-e050-4e18-885d-52a560c5deb7" path="/var/lib/kubelet/pods/44780614-e050-4e18-885d-52a560c5deb7/volumes" Oct 29 23:05:59.308248 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.308182 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:05:59.355084 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.351522 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:05:59.424761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.418130 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm\" (UniqueName: \"kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm\") pod \"612a9ec7-d897-49a0-b224-aeae8495961d\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " Oct 29 23:05:59.424761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.418219 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj\" (UniqueName: \"kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj\") pod \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " Oct 29 23:05:59.424761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.418348 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume\") pod \"612a9ec7-d897-49a0-b224-aeae8495961d\" (UID: \"612a9ec7-d897-49a0-b224-aeae8495961d\") " Oct 29 23:05:59.424761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.418387 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca\") pod \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " Oct 29 23:05:59.424761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.418438 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca\") pod \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\" (UID: \"b6d9b164-4ce6-4cac-8216-233a564ab6cd\") " Oct 29 23:05:59.425389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.425351 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca" pod "b6d9b164-4ce6-4cac-8216-233a564ab6cd" (UID: "b6d9b164-4ce6-4cac-8216-233a564ab6cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:59.426124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.425809 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume" pod "612a9ec7-d897-49a0-b224-aeae8495961d" (UID: "612a9ec7-d897-49a0-b224-aeae8495961d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:59.433513 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.433473 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm" pod "612a9ec7-d897-49a0-b224-aeae8495961d" (UID: "612a9ec7-d897-49a0-b224-aeae8495961d"). InnerVolumeSpecName "kube-api-access-b4vfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:59.433766 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.433735 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca" pod "b6d9b164-4ce6-4cac-8216-233a564ab6cd" (UID: "b6d9b164-4ce6-4cac-8216-233a564ab6cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:59.433987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.433960 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj" pod "b6d9b164-4ce6-4cac-8216-233a564ab6cd" (UID: "b6d9b164-4ce6-4cac-8216-233a564ab6cd"). InnerVolumeSpecName "kube-api-access-wqkwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:59.520709 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.520674 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqkwj\" (UniqueName: \"kubernetes.io/projected/b6d9b164-4ce6-4cac-8216-233a564ab6cd-kube-api-access-wqkwj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.520877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.520737 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612a9ec7-d897-49a0-b224-aeae8495961d-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.520877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.520751 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.520877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.520761 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9b164-4ce6-4cac-8216-233a564ab6cd-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.520877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.520769 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4vfm\" (UniqueName: \"kubernetes.io/projected/612a9ec7-d897-49a0-b224-aeae8495961d-kube-api-access-b4vfm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.563134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.563049 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6"] Oct 29 23:05:59.563885 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:05:59.563850 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c74e55-9cts6"] Oct 29 23:06:00.135817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.135789 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" Oct 29 23:06:00.135817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.135810 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs" event={"ID":"b6d9b164-4ce6-4cac-8216-233a564ab6cd","Type":"ContainerDied","Data":"2820f6e8e097fb9f4cbba81ae0e02306b1fbd6b8b0ee88e37d272e6c70505ff6"} Oct 29 23:06:00.136408 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.135888 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2820f6e8e097fb9f4cbba81ae0e02306b1fbd6b8b0ee88e37d272e6c70505ff6" Oct 29 23:06:00.137335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.137314 5367 generic.go:356] "Generic (PLEG): container finished" podID="612a9ec7-d897-49a0-b224-aeae8495961d" containerID="26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7" exitCode=0 Oct 29 23:06:00.137460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.137372 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" Oct 29 23:06:00.137460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.137399 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" event={"ID":"612a9ec7-d897-49a0-b224-aeae8495961d","Type":"ContainerDied","Data":"26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7"} Oct 29 23:06:00.137460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.137453 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5" event={"ID":"612a9ec7-d897-49a0-b224-aeae8495961d","Type":"ContainerDied","Data":"ab49bffd518edd181fdc943bcf38b321aa11fad50bb21247aeb15ab9c5e571d7"} Oct 29 23:06:00.137595 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.137485 5367 scope.go:122] "RemoveContainer" containerID="26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7" Oct 29 23:06:00.149799 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.149783 5367 scope.go:122] "RemoveContainer" containerID="26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7" Oct 29 23:06:00.150051 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:00.150033 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7\": container with ID starting with 26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7 not found: ID does not exist" containerID="26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7" Oct 29 23:06:00.150103 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.150059 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7"} err="failed to get container status \"26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7\": rpc error: code = NotFound desc = could not find container \"26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7\": container with ID starting with 26b037b7a8a71dfbdf6a733ff63dd97e72e1a2fd5de6c38cd3a5a54ac44fbaa7 not found: ID does not exist" Oct 29 23:06:00.202045 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.202011 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5"] Oct 29 23:06:00.205440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.205414 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c74e55-69b95b5c99-c4sv5"] Oct 29 23:06:00.740386 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.740360 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 23:06:00.863762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.863731 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.863951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.863788 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.863951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.863841 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.863955 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.863995 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864015 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864038 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864085 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864366 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864124 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket\") pod \"2e312dd8-3618-4b93-9fc6-321f2460aa77\" (UID: \"2e312dd8-3618-4b93-9fc6-321f2460aa77\") " Oct 29 23:06:00.864366 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864175 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.864486 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864394 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.864486 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864415 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:00.864660 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864636 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.864770 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864653 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.864823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864765 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e312dd8-3618-4b93-9fc6-321f2460aa77-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.864823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864789 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.864823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.864801 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.866109 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.866088 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "kube-api-access-wsqvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:00.866208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.866188 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.866440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.866423 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:00.866511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.866496 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo" pod "2e312dd8-3618-4b93-9fc6-321f2460aa77" (UID: "2e312dd8-3618-4b93-9fc6-321f2460aa77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:00.966515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.966402 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.966515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.966471 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.966515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.966482 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.966515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.966490 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.966515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.966500 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e312dd8-3618-4b93-9fc6-321f2460aa77-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.966515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:00.966511 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsqvz\" (UniqueName: \"kubernetes.io/projected/2e312dd8-3618-4b93-9fc6-321f2460aa77-kube-api-access-wsqvz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.147332 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.147297 5367 generic.go:356] "Generic (PLEG): container finished" podID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerID="13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6" exitCode=0 Oct 29 23:06:01.147825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.147352 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" Oct 29 23:06:01.147825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.147367 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" event={"ID":"2e312dd8-3618-4b93-9fc6-321f2460aa77","Type":"ContainerDied","Data":"13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6"} Oct 29 23:06:01.147825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.147417 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv" event={"ID":"2e312dd8-3618-4b93-9fc6-321f2460aa77","Type":"ContainerDied","Data":"d3d465ef6f491df70968fbf7124c55781e44b57e0be47a70b80532fe7a9d4216"} Oct 29 23:06:01.147825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.147446 5367 scope.go:122] "RemoveContainer" containerID="13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6" Oct 29 23:06:01.159087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.159065 5367 scope.go:122] "RemoveContainer" containerID="b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71" Oct 29 23:06:01.169200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.169178 5367 scope.go:122] "RemoveContainer" containerID="13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6" Oct 29 23:06:01.169472 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:01.169452 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6\": container with ID starting with 13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6 not found: ID does not exist" containerID="13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6" Oct 29 23:06:01.169522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.169482 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6"} err="failed to get container status \"13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6\": rpc error: code = NotFound desc = could not find container \"13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6\": container with ID starting with 13f193ce93219b2e3c5d461daa1118dbc710742e6a901dd1226adb6e65e572c6 not found: ID does not exist" Oct 29 23:06:01.169522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.169504 5367 scope.go:122] "RemoveContainer" containerID="b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71" Oct 29 23:06:01.169765 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:01.169748 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71\": container with ID starting with b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71 not found: ID does not exist" containerID="b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71" Oct 29 23:06:01.169812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.169770 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71"} err="failed to get container status \"b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71\": rpc error: code = NotFound desc = could not find container \"b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71\": container with ID starting with b8b8231ae880da27266500d35293080980c742094ab481e58289337466ae6e71 not found: ID does not exist" Oct 29 23:06:01.205254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.205224 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv"] Oct 29 23:06:01.222036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.221972 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612a9ec7-d897-49a0-b224-aeae8495961d" path="/var/lib/kubelet/pods/612a9ec7-d897-49a0-b224-aeae8495961d/volumes" Oct 29 23:06:01.222491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.222475 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758ed51f-570b-4598-bae8-5492736024be" path="/var/lib/kubelet/pods/758ed51f-570b-4598-bae8-5492736024be/volumes" Oct 29 23:06:01.222807 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.222793 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-cmxgv"] Oct 29 23:06:01.744036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.744002 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9"] Oct 29 23:06:01.744249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:01.744223 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" containerID="cri-o://8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf" gracePeriod=30 Oct 29 23:06:02.231303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231272 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7"] Oct 29 23:06:02.231848 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231832 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231849 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231871 5367 cpu_manager.go:414] "containerMap: removing container" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231876 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231884 5367 cpu_manager.go:414] "containerMap: removing container" podUID="612a9ec7-d897-49a0-b224-aeae8495961d" containerName="payload-pre-processing" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231889 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612a9ec7-d897-49a0-b224-aeae8495961d" containerName="payload-pre-processing" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231896 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9b164-4ce6-4cac-8216-233a564ab6cd" containerName="revoke-keys" Oct 29 23:06:02.231908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231901 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9b164-4ce6-4cac-8216-233a564ab6cd" containerName="revoke-keys" Oct 29 23:06:02.232132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231919 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" Oct 29 23:06:02.232132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231924 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" Oct 29 23:06:02.232132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.231998 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="44780614-e050-4e18-885d-52a560c5deb7" containerName="istio-proxy" Oct 29 23:06:02.232132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.232006 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="612a9ec7-d897-49a0-b224-aeae8495961d" containerName="payload-pre-processing" Oct 29 23:06:02.232132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.232013 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9b164-4ce6-4cac-8216-233a564ab6cd" containerName="revoke-keys" Oct 29 23:06:02.232132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.232021 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" Oct 29 23:06:02.237072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.237053 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.254219 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.254192 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:06:02.254348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.254322 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:06:02.267758 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.267735 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:02.267954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.267937 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:02.271708 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.271673 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7"] Oct 29 23:06:02.382052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.382015 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.382242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.382155 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5mv\" (UniqueName: \"kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.382901 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.382872 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.483862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.483782 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.483862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.483848 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.484116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.483904 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5mv\" (UniqueName: \"kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.484405 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.484387 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.486176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.486157 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.504364 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.504340 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5mv\" (UniqueName: \"kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.552218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.552196 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:02.857575 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.857539 5367 patch_prober.go:29] interesting pod/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:02.857751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.857604 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:02.861464 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:02.861437 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7"] Oct 29 23:06:02.862383 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:02.862356 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f077aed_78a4_4ff6_823d_c37a739d90a5.slice/crio-196e3445fc77dc9ba2205a01ab2776d8713e4b964de898eb50d46b4d272036dd WatchSource:0}: Error finding container 196e3445fc77dc9ba2205a01ab2776d8713e4b964de898eb50d46b4d272036dd: Status 404 returned error can't find the container with id 196e3445fc77dc9ba2205a01ab2776d8713e4b964de898eb50d46b4d272036dd Oct 29 23:06:03.171766 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:03.171736 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" event={"ID":"4f077aed-78a4-4ff6-823d-c37a739d90a5","Type":"ContainerStarted","Data":"196e3445fc77dc9ba2205a01ab2776d8713e4b964de898eb50d46b4d272036dd"} Oct 29 23:06:03.221992 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:03.221955 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" path="/var/lib/kubelet/pods/2e312dd8-3618-4b93-9fc6-321f2460aa77/volumes" Oct 29 23:06:04.018488 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:04.018381 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp"] Oct 29 23:06:04.018984 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:04.018603 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" containerID="cri-o://57122ff1d1a88e44b21922d7ce44504d0748b90c60eb02b07842f7ab58e19793" gracePeriod=30 Oct 29 23:06:04.181091 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:04.181053 5367 generic.go:356] "Generic (PLEG): container finished" podID="4f077aed-78a4-4ff6-823d-c37a739d90a5" containerID="07a02ecc5e7359a972c9be5c9ab0961f9f13fb7306330ae9451403813d7fa773" exitCode=0 Oct 29 23:06:04.181314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:04.181136 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" event={"ID":"4f077aed-78a4-4ff6-823d-c37a739d90a5","Type":"ContainerDied","Data":"07a02ecc5e7359a972c9be5c9ab0961f9f13fb7306330ae9451403813d7fa773"} Oct 29 23:06:05.370655 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.370632 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:05.426463 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.426408 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca\") pod \"4f077aed-78a4-4ff6-823d-c37a739d90a5\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " Oct 29 23:06:05.426610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.426550 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv\" (UniqueName: \"kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv\") pod \"4f077aed-78a4-4ff6-823d-c37a739d90a5\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " Oct 29 23:06:05.426680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.426660 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca\") pod \"4f077aed-78a4-4ff6-823d-c37a739d90a5\" (UID: \"4f077aed-78a4-4ff6-823d-c37a739d90a5\") " Oct 29 23:06:05.427049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.427021 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca" pod "4f077aed-78a4-4ff6-823d-c37a739d90a5" (UID: "4f077aed-78a4-4ff6-823d-c37a739d90a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:05.427235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.427215 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:05.428746 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.428721 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca" pod "4f077aed-78a4-4ff6-823d-c37a739d90a5" (UID: "4f077aed-78a4-4ff6-823d-c37a739d90a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:05.428927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.428907 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv" pod "4f077aed-78a4-4ff6-823d-c37a739d90a5" (UID: "4f077aed-78a4-4ff6-823d-c37a739d90a5"). InnerVolumeSpecName "kube-api-access-5k5mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:05.528520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.528438 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f077aed-78a4-4ff6-823d-c37a739d90a5-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:05.528520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:05.528465 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k5mv\" (UniqueName: \"kubernetes.io/projected/4f077aed-78a4-4ff6-823d-c37a739d90a5-kube-api-access-5k5mv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:06.165018 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:06.164981 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:06.165195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:06.165041 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:06.198076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:06.198049 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" Oct 29 23:06:06.198204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:06.198087 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7" event={"ID":"4f077aed-78a4-4ff6-823d-c37a739d90a5","Type":"ContainerDied","Data":"196e3445fc77dc9ba2205a01ab2776d8713e4b964de898eb50d46b4d272036dd"} Oct 29 23:06:06.198204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:06.198134 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="196e3445fc77dc9ba2205a01ab2776d8713e4b964de898eb50d46b4d272036dd" Oct 29 23:06:06.482398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:06.482312 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb"] Oct 29 23:06:07.018504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.018421 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:06:07.051834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.051807 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.051993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.051853 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.051993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.051876 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.051993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.051903 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.051993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.051971 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.052191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052097 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.052191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052178 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.052305 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052221 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:07.052305 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052246 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.052305 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052275 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo\") pod \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\" (UID: \"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b\") " Oct 29 23:06:07.052602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052277 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:07.052602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052361 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:07.052602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052468 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:07.052602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052559 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:07.052942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052853 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.052942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052875 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.052942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052888 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.052942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052902 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.052942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.052916 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.054205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.054175 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:07.054302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.054184 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:07.054302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.054224 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "kube-api-access-hvj4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:07.054302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.054234 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo" pod "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" (UID: "7ff303e1-2b83-4733-b3f2-ed0fb7058a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:07.138730 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.138676 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn"] Oct 29 23:06:07.139005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.138978 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" containerID="cri-o://90b201e3b953fcf6b08a5c379a8b668a52100f0a67f1499592ed14841c76910a" gracePeriod=30 Oct 29 23:06:07.154885 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.154848 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.154885 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.154888 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.155087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.154903 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvj4j\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-kube-api-access-hvj4j\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.155087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.154919 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:07.207894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.207857 5367 generic.go:356] "Generic (PLEG): container finished" podID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerID="8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf" exitCode=0 Oct 29 23:06:07.208073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.207917 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" event={"ID":"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b","Type":"ContainerDied","Data":"8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf"} Oct 29 23:06:07.208073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.207923 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" Oct 29 23:06:07.208073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.207994 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9" event={"ID":"7ff303e1-2b83-4733-b3f2-ed0fb7058a3b","Type":"ContainerDied","Data":"478c09d3c70cb013f020d6b42652bfba70b9a806aaaa0b5cb1da8d5c8800bc22"} Oct 29 23:06:07.208073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.208027 5367 scope.go:122] "RemoveContainer" containerID="8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf" Oct 29 23:06:07.220144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.220123 5367 scope.go:122] "RemoveContainer" containerID="170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55" Oct 29 23:06:07.231825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.231777 5367 scope.go:122] "RemoveContainer" containerID="8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf" Oct 29 23:06:07.232116 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:07.232089 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf\": container with ID starting with 8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf not found: ID does not exist" containerID="8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf" Oct 29 23:06:07.232222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.232129 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf"} err="failed to get container status \"8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf\": rpc error: code = NotFound desc = could not find container \"8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf\": container with ID starting with 8d187e23659107e2d687f04448a253fb89b4aae70dcd2d251b594b47a9bc52bf not found: ID does not exist" Oct 29 23:06:07.232222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.232151 5367 scope.go:122] "RemoveContainer" containerID="170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55" Oct 29 23:06:07.232663 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:07.232627 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55\": container with ID starting with 170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55 not found: ID does not exist" containerID="170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55" Oct 29 23:06:07.232886 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.232657 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55"} err="failed to get container status \"170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55\": rpc error: code = NotFound desc = could not find container \"170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55\": container with ID starting with 170c7a5b7deada322e6419a0541c3e62a26fe358e9ccaaf931a282fba34e8e55 not found: ID does not exist" Oct 29 23:06:07.300185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.300082 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9"] Oct 29 23:06:07.306167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.301345 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eddb03-openshift-default-5bcd44f6df-mpmx9"] Oct 29 23:06:07.702718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.702669 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr"] Oct 29 23:06:07.703230 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703215 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4f077aed-78a4-4ff6-823d-c37a739d90a5" containerName="revoke-keys" Oct 29 23:06:07.703272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703231 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f077aed-78a4-4ff6-823d-c37a739d90a5" containerName="revoke-keys" Oct 29 23:06:07.703272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703243 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" Oct 29 23:06:07.703272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703249 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" Oct 29 23:06:07.703378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703336 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f077aed-78a4-4ff6-823d-c37a739d90a5" containerName="revoke-keys" Oct 29 23:06:07.703378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703346 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" Oct 29 23:06:07.703378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.703352 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e312dd8-3618-4b93-9fc6-321f2460aa77" containerName="istio-proxy" Oct 29 23:06:07.708263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.708238 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.724646 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.724586 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:06:07.724872 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.724659 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:06:07.726574 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.726549 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr"] Oct 29 23:06:07.747842 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.747815 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:07.748010 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.747973 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:07.771608 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.771577 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.771822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.771802 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.771869 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.771857 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4746d\" (UniqueName: \"kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.872555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.872523 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.872780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.872648 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.872780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.872742 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4746d\" (UniqueName: \"kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.873362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.873337 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.874972 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.874945 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:07.896921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:07.896892 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4746d\" (UniqueName: \"kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:08.023397 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:08.023317 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:08.327184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:08.327156 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr"] Oct 29 23:06:08.327986 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:08.327963 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fee8bbd_211a_4abc_8d85_fb7cc62219c0.slice/crio-30945d2e776bc4cd5ed1fdc1fa5f4748bcdc973bf02cd696a4e507d6706ebe76 WatchSource:0}: Error finding container 30945d2e776bc4cd5ed1fdc1fa5f4748bcdc973bf02cd696a4e507d6706ebe76: Status 404 returned error can't find the container with id 30945d2e776bc4cd5ed1fdc1fa5f4748bcdc973bf02cd696a4e507d6706ebe76 Oct 29 23:06:09.224427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.224395 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" path="/var/lib/kubelet/pods/7ff303e1-2b83-4733-b3f2-ed0fb7058a3b/volumes" Oct 29 23:06:09.233040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.233009 5367 generic.go:356] "Generic (PLEG): container finished" podID="0fee8bbd-211a-4abc-8d85-fb7cc62219c0" containerID="94d984c3b0ec39ee5a89f25b2e63fe9e8f7885b23ada48b33b1e54424c08dff6" exitCode=0 Oct 29 23:06:09.233185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.233111 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" event={"ID":"0fee8bbd-211a-4abc-8d85-fb7cc62219c0","Type":"ContainerDied","Data":"94d984c3b0ec39ee5a89f25b2e63fe9e8f7885b23ada48b33b1e54424c08dff6"} Oct 29 23:06:09.233185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.233166 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" event={"ID":"0fee8bbd-211a-4abc-8d85-fb7cc62219c0","Type":"ContainerStarted","Data":"30945d2e776bc4cd5ed1fdc1fa5f4748bcdc973bf02cd696a4e507d6706ebe76"} Oct 29 23:06:09.235388 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.235366 5367 generic.go:356] "Generic (PLEG): container finished" podID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerID="57122ff1d1a88e44b21922d7ce44504d0748b90c60eb02b07842f7ab58e19793" exitCode=0 Oct 29 23:06:09.235535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.235459 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" event={"ID":"be0a4407-0222-426a-bc2e-49c63ac7f58a","Type":"ContainerDied","Data":"57122ff1d1a88e44b21922d7ce44504d0748b90c60eb02b07842f7ab58e19793"} Oct 29 23:06:09.298040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.297982 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 23:06:09.385574 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385545 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385589 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385613 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385644 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385715 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385757 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385822 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.385961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385889 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:09.385961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.385906 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.386193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386030 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8\") pod \"be0a4407-0222-426a-bc2e-49c63ac7f58a\" (UID: \"be0a4407-0222-426a-bc2e-49c63ac7f58a\") " Oct 29 23:06:09.386193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386155 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:09.386433 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386405 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:09.386581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386535 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.386654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386600 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.386654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386466 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:09.386654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.386580 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:09.388151 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.388128 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:09.388270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.388252 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:09.388447 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.388430 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "kube-api-access-vxjg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:09.388493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.388433 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo" pod "be0a4407-0222-426a-bc2e-49c63ac7f58a" (UID: "be0a4407-0222-426a-bc2e-49c63ac7f58a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:09.487714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487672 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a4407-0222-426a-bc2e-49c63ac7f58a-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.487714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487717 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxjg8\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-kube-api-access-vxjg8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.487935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487734 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.487935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487745 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.487935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487754 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.487935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487761 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:09.487935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:09.487771 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a4407-0222-426a-bc2e-49c63ac7f58a-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:10.243867 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.243835 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" Oct 29 23:06:10.244298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.243863 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp" event={"ID":"be0a4407-0222-426a-bc2e-49c63ac7f58a","Type":"ContainerDied","Data":"08a9bd2c783a3614aca74eaf0429de2648e77b9ae4ebb70adf17ffe6aa8294f2"} Oct 29 23:06:10.244298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.243932 5367 scope.go:122] "RemoveContainer" containerID="57122ff1d1a88e44b21922d7ce44504d0748b90c60eb02b07842f7ab58e19793" Oct 29 23:06:10.316142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.316113 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp"] Oct 29 23:06:10.329598 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.329570 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-45vtp"] Oct 29 23:06:10.387191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.387166 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:10.398232 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.398207 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca\") pod \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " Oct 29 23:06:10.398350 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.398298 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca\") pod \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " Oct 29 23:06:10.398395 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.398349 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d\" (UniqueName: \"kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d\") pod \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\" (UID: \"0fee8bbd-211a-4abc-8d85-fb7cc62219c0\") " Oct 29 23:06:10.398536 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.398514 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca" pod "0fee8bbd-211a-4abc-8d85-fb7cc62219c0" (UID: "0fee8bbd-211a-4abc-8d85-fb7cc62219c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:10.398915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.398898 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:10.400323 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.400300 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d" pod "0fee8bbd-211a-4abc-8d85-fb7cc62219c0" (UID: "0fee8bbd-211a-4abc-8d85-fb7cc62219c0"). InnerVolumeSpecName "kube-api-access-4746d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:10.400396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.400364 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca" pod "0fee8bbd-211a-4abc-8d85-fb7cc62219c0" (UID: "0fee8bbd-211a-4abc-8d85-fb7cc62219c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:10.500069 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.499992 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:10.500069 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:10.500021 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4746d\" (UniqueName: \"kubernetes.io/projected/0fee8bbd-211a-4abc-8d85-fb7cc62219c0-kube-api-access-4746d\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:11.224487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:11.224449 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" path="/var/lib/kubelet/pods/be0a4407-0222-426a-bc2e-49c63ac7f58a/volumes" Oct 29 23:06:11.254291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:11.254261 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" event={"ID":"0fee8bbd-211a-4abc-8d85-fb7cc62219c0","Type":"ContainerDied","Data":"30945d2e776bc4cd5ed1fdc1fa5f4748bcdc973bf02cd696a4e507d6706ebe76"} Oct 29 23:06:11.254612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:11.254313 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30945d2e776bc4cd5ed1fdc1fa5f4748bcdc973bf02cd696a4e507d6706ebe76" Oct 29 23:06:11.254612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:11.254274 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr" Oct 29 23:06:11.482850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:11.482762 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerName="payload-processing" containerID="cri-o://bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097" gracePeriod=25 Oct 29 23:06:12.266198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.266171 5367 generic.go:356] "Generic (PLEG): container finished" podID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerID="90b201e3b953fcf6b08a5c379a8b668a52100f0a67f1499592ed14841c76910a" exitCode=0 Oct 29 23:06:12.266597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.266242 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" event={"ID":"beee6908-14c0-4bde-ad79-ee142dd07a29","Type":"ContainerDied","Data":"90b201e3b953fcf6b08a5c379a8b668a52100f0a67f1499592ed14841c76910a"} Oct 29 23:06:12.266597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.266335 5367 scope.go:122] "RemoveContainer" containerID="b85aa3e689949d24345d952346ac74baf4e193e6a3b87eae30e8790363488d64" Oct 29 23:06:12.406519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.406462 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:06:12.420455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420420 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420468 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420498 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420525 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420563 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420676 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420692 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.420884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420733 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420815 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.420884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420824 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.420884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420840 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.421150 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.420898 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert\") pod \"beee6908-14c0-4bde-ad79-ee142dd07a29\" (UID: \"beee6908-14c0-4bde-ad79-ee142dd07a29\") " Oct 29 23:06:12.421150 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421049 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.421325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421307 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:12.421485 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421469 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.421557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421490 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.421557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421504 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.421557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421516 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.421557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.421531 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beee6908-14c0-4bde-ad79-ee142dd07a29-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.422952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.422924 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:12.423063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.422968 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "kube-api-access-qv9lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:12.423218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.423194 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:12.423346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.423285 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy" pod "beee6908-14c0-4bde-ad79-ee142dd07a29" (UID: "beee6908-14c0-4bde-ad79-ee142dd07a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.522319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.522289 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.522319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.522317 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9lm\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-kube-api-access-qv9lm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.522319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.522328 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.522583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:12.522336 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beee6908-14c0-4bde-ad79-ee142dd07a29-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:13.275288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:13.275259 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" Oct 29 23:06:13.275677 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:13.275295 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn" event={"ID":"beee6908-14c0-4bde-ad79-ee142dd07a29","Type":"ContainerDied","Data":"3859de540e904ce892a83294fb2eea2f099dd8e584f30dd9f76f2fcdb7db6a50"} Oct 29 23:06:13.275677 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:13.275349 5367 scope.go:122] "RemoveContainer" containerID="90b201e3b953fcf6b08a5c379a8b668a52100f0a67f1499592ed14841c76910a" Oct 29 23:06:13.364752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:13.364715 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn"] Oct 29 23:06:13.369088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:13.369061 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c74e55-openshift-default-958874d7-hx8mn"] Oct 29 23:06:15.221874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:15.221844 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" path="/var/lib/kubelet/pods/beee6908-14c0-4bde-ad79-ee142dd07a29/volumes" Oct 29 23:06:16.866076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:16.866042 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7"] Oct 29 23:06:16.878389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:16.878361 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a15f4-k4pl7"] Oct 29 23:06:17.221441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:17.221349 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f077aed-78a4-4ff6-823d-c37a739d90a5" path="/var/lib/kubelet/pods/4f077aed-78a4-4ff6-823d-c37a739d90a5/volumes" Oct 29 23:06:18.150526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:18.150492 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj"] Oct 29 23:06:18.184830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:18.184798 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887"] Oct 29 23:06:18.817911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:18.817881 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs"] Oct 29 23:06:18.820641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:18.820619 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6bb06a-j57hs"] Oct 29 23:06:19.220863 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:19.220789 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9b164-4ce6-4cac-8216-233a564ab6cd" path="/var/lib/kubelet/pods/b6d9b164-4ce6-4cac-8216-233a564ab6cd/volumes" Oct 29 23:06:20.708114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:20.708069 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:22.330900 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:22.330867 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:06:23.150853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:23.150805 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerName="payload-processing" containerID="cri-o://7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51" gracePeriod=25 Oct 29 23:06:23.185345 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:23.185300 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerName="payload-pre-processing" containerID="cri-o://6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1" gracePeriod=25 Oct 29 23:06:24.356590 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.356559 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln"] Oct 29 23:06:24.357075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357062 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357076 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357092 5367 cpu_manager.go:414] "containerMap: removing container" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357098 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357105 5367 cpu_manager.go:414] "containerMap: removing container" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357111 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357122 5367 cpu_manager.go:414] "containerMap: removing container" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357126 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" Oct 29 23:06:24.357134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357136 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0fee8bbd-211a-4abc-8d85-fb7cc62219c0" containerName="revoke-keys" Oct 29 23:06:24.357377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357140 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fee8bbd-211a-4abc-8d85-fb7cc62219c0" containerName="revoke-keys" Oct 29 23:06:24.357377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357233 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" Oct 29 23:06:24.357377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357240 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="beee6908-14c0-4bde-ad79-ee142dd07a29" containerName="istio-proxy" Oct 29 23:06:24.357377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357247 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0a4407-0222-426a-bc2e-49c63ac7f58a" containerName="istio-proxy" Oct 29 23:06:24.357377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357254 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fee8bbd-211a-4abc-8d85-fb7cc62219c0" containerName="revoke-keys" Oct 29 23:06:24.357377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.357267 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff303e1-2b83-4733-b3f2-ed0fb7058a3b" containerName="istio-proxy" Oct 29 23:06:24.362106 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.362089 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.395542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.395509 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:24.395542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.395509 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:06:24.395757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.395525 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:06:24.395757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.395659 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:24.399568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.399543 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln"] Oct 29 23:06:24.456283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.456248 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hffg\" (UniqueName: \"kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.456456 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.456287 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.456456 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.456361 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.557326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.557281 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hffg\" (UniqueName: \"kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.557326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.557332 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.557558 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.557376 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.558107 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.558073 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.559682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.559664 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.578776 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.578745 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hffg\" (UniqueName: \"kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.675663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.675566 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:24.978359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:24.978333 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln"] Oct 29 23:06:24.979084 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:24.979053 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad2e9e6_77df_4675_adc9_b5d0613786f9.slice/crio-8f00ffa85732113a0f1af8b302cafca39f60c1005d3051851a2884f0da534273 WatchSource:0}: Error finding container 8f00ffa85732113a0f1af8b302cafca39f60c1005d3051851a2884f0da534273: Status 404 returned error can't find the container with id 8f00ffa85732113a0f1af8b302cafca39f60c1005d3051851a2884f0da534273 Oct 29 23:06:25.366435 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:25.366401 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" event={"ID":"8ad2e9e6-77df-4675-adc9-b5d0613786f9","Type":"ContainerStarted","Data":"db85d9e9d0b3e22ee1458cfd00d5502b552b0e04e5dc9510db837f35c3f6e6b3"} Oct 29 23:06:25.366908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:25.366461 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" event={"ID":"8ad2e9e6-77df-4675-adc9-b5d0613786f9","Type":"ContainerStarted","Data":"8f00ffa85732113a0f1af8b302cafca39f60c1005d3051851a2884f0da534273"} Oct 29 23:06:25.410942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:25.410887 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" podStartSLOduration=1.410873644 podStartE2EDuration="1.410873644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:25.407443411 +0000 UTC m=+3052.301518333" watchObservedRunningTime="2026-10-29 23:06:25.410873644 +0000 UTC m=+3052.304948850" Oct 29 23:06:26.139668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.139634 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp"] Oct 29 23:06:26.139919 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.139869 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" containerID="cri-o://8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764" gracePeriod=30 Oct 29 23:06:26.375533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.375494 5367 generic.go:356] "Generic (PLEG): container finished" podID="8ad2e9e6-77df-4675-adc9-b5d0613786f9" containerID="db85d9e9d0b3e22ee1458cfd00d5502b552b0e04e5dc9510db837f35c3f6e6b3" exitCode=0 Oct 29 23:06:26.376006 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.375587 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" event={"ID":"8ad2e9e6-77df-4675-adc9-b5d0613786f9","Type":"ContainerDied","Data":"db85d9e9d0b3e22ee1458cfd00d5502b552b0e04e5dc9510db837f35c3f6e6b3"} Oct 29 23:06:26.797811 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.797776 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k"] Oct 29 23:06:26.802244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.802219 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.821747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.821685 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k"] Oct 29 23:06:26.884114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.884082 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k882d\" (UniqueName: \"kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.884293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.884189 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.884293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.884234 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.985855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.985824 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k882d\" (UniqueName: \"kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.986039 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.985893 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.986039 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.985931 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.986622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.986591 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:26.988379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:26.988359 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:27.008340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.008313 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k882d\" (UniqueName: \"kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d\") pod \"maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:27.116249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.116199 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:27.417258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.417224 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k"] Oct 29 23:06:27.418450 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:27.418404 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0572a5_4a37_46d7_8229_5ac6a32a0fcd.slice/crio-90f00993a8cf77bdb4ad59eb8080f135943b5a70556cd51679f844a6656fed69 WatchSource:0}: Error finding container 90f00993a8cf77bdb4ad59eb8080f135943b5a70556cd51679f844a6656fed69: Status 404 returned error can't find the container with id 90f00993a8cf77bdb4ad59eb8080f135943b5a70556cd51679f844a6656fed69 Oct 29 23:06:27.509075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.509051 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:27.593738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.593695 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg\" (UniqueName: \"kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg\") pod \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " Oct 29 23:06:27.593936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.593794 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca\") pod \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " Oct 29 23:06:27.593936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.593920 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca\") pod \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\" (UID: \"8ad2e9e6-77df-4675-adc9-b5d0613786f9\") " Oct 29 23:06:27.594198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.594173 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca" pod "8ad2e9e6-77df-4675-adc9-b5d0613786f9" (UID: "8ad2e9e6-77df-4675-adc9-b5d0613786f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:27.594403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.594386 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:27.654522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.654425 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca" pod "8ad2e9e6-77df-4675-adc9-b5d0613786f9" (UID: "8ad2e9e6-77df-4675-adc9-b5d0613786f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:27.654522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.654454 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg" pod "8ad2e9e6-77df-4675-adc9-b5d0613786f9" (UID: "8ad2e9e6-77df-4675-adc9-b5d0613786f9"). InnerVolumeSpecName "kube-api-access-2hffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:27.696002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.695966 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad2e9e6-77df-4675-adc9-b5d0613786f9-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:27.696002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:27.695995 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hffg\" (UniqueName: \"kubernetes.io/projected/8ad2e9e6-77df-4675-adc9-b5d0613786f9-kube-api-access-2hffg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:28.392770 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.392733 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" event={"ID":"8ad2e9e6-77df-4675-adc9-b5d0613786f9","Type":"ContainerDied","Data":"8f00ffa85732113a0f1af8b302cafca39f60c1005d3051851a2884f0da534273"} Oct 29 23:06:28.392924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.392804 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f00ffa85732113a0f1af8b302cafca39f60c1005d3051851a2884f0da534273" Oct 29 23:06:28.392924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.392746 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln" Oct 29 23:06:28.394757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.394731 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" event={"ID":"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd","Type":"ContainerStarted","Data":"504f22e9e017e9a96f422599a761106e655d621c27b91aa39650254520e24066"} Oct 29 23:06:28.394876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.394778 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" event={"ID":"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd","Type":"ContainerStarted","Data":"90f00993a8cf77bdb4ad59eb8080f135943b5a70556cd51679f844a6656fed69"} Oct 29 23:06:28.441554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.441496 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" podStartSLOduration=2.4414826 podStartE2EDuration="2.4414826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:28.437784094 +0000 UTC m=+3055.331859013" watchObservedRunningTime="2026-10-29 23:06:28.4414826 +0000 UTC m=+3055.335557516" Oct 29 23:06:28.790520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.790414 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr"] Oct 29 23:06:28.793366 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:28.793341 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36b46d-5m6vr"] Oct 29 23:06:29.225316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:29.225276 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fee8bbd-211a-4abc-8d85-fb7cc62219c0" path="/var/lib/kubelet/pods/0fee8bbd-211a-4abc-8d85-fb7cc62219c0/volumes" Oct 29 23:06:29.403806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:29.403774 5367 generic.go:356] "Generic (PLEG): container finished" podID="2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" containerID="504f22e9e017e9a96f422599a761106e655d621c27b91aa39650254520e24066" exitCode=0 Oct 29 23:06:29.403964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:29.403856 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" event={"ID":"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd","Type":"ContainerDied","Data":"504f22e9e017e9a96f422599a761106e655d621c27b91aa39650254520e24066"} Oct 29 23:06:29.496378 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:29.496298 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4"] Oct 29 23:06:29.496854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:29.496487 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" containerID="cri-o://b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f" gracePeriod=30 Oct 29 23:06:30.548062 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.548033 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:30.634363 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.634333 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d\" (UniqueName: \"kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d\") pod \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " Oct 29 23:06:30.634542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.634432 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca\") pod \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " Oct 29 23:06:30.634542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.634478 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca\") pod \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\" (UID: \"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd\") " Oct 29 23:06:30.634992 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.634963 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca" pod "2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" (UID: "2d0572a5-4a37-46d7-8229-5ac6a32a0fcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:30.636591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.636561 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca" pod "2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" (UID: "2d0572a5-4a37-46d7-8229-5ac6a32a0fcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:30.636680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.636600 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d" pod "2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" (UID: "2d0572a5-4a37-46d7-8229-5ac6a32a0fcd"). InnerVolumeSpecName "kube-api-access-k882d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:30.707987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.707907 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:30.735783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.735752 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k882d\" (UniqueName: \"kubernetes.io/projected/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-kube-api-access-k882d\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:30.735930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.735811 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:30.735930 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:30.735827 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.387906 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.387879 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 23:06:31.420640 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.420562 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" Oct 29 23:06:31.420640 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.420606 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k" event={"ID":"2d0572a5-4a37-46d7-8229-5ac6a32a0fcd","Type":"ContainerDied","Data":"90f00993a8cf77bdb4ad59eb8080f135943b5a70556cd51679f844a6656fed69"} Oct 29 23:06:31.420866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.420658 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f00993a8cf77bdb4ad59eb8080f135943b5a70556cd51679f844a6656fed69" Oct 29 23:06:31.422113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.422087 5367 generic.go:356] "Generic (PLEG): container finished" podID="bd284d9a-851f-48c8-a365-b456bffc3149" containerID="8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764" exitCode=0 Oct 29 23:06:31.422240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.422130 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" Oct 29 23:06:31.422240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.422173 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" event={"ID":"bd284d9a-851f-48c8-a365-b456bffc3149","Type":"ContainerDied","Data":"8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764"} Oct 29 23:06:31.422240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.422231 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp" event={"ID":"bd284d9a-851f-48c8-a365-b456bffc3149","Type":"ContainerDied","Data":"ec49c17bc5ab83965268407a0f944c5867592efed7df76e5463ab34e7cb87c95"} Oct 29 23:06:31.422374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.422264 5367 scope.go:122] "RemoveContainer" containerID="8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764" Oct 29 23:06:31.433296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.433279 5367 scope.go:122] "RemoveContainer" containerID="8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764" Oct 29 23:06:31.433582 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:31.433563 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764\": container with ID starting with 8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764 not found: ID does not exist" containerID="8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764" Oct 29 23:06:31.433680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.433590 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764"} err="failed to get container status \"8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764\": rpc error: code = NotFound desc = could not find container \"8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764\": container with ID starting with 8cae1c36be455c5f9c03356af20567ecb537117c5a58e7fdb683dbfbed5a5764 not found: ID does not exist" Oct 29 23:06:31.443992 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.443968 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444034 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444073 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444109 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444135 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444151 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444174 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444343 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.444436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444391 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:31.444436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444401 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.444436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444402 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444744 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444455 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data\") pod \"bd284d9a-851f-48c8-a365-b456bffc3149\" (UID: \"bd284d9a-851f-48c8-a365-b456bffc3149\") " Oct 29 23:06:31.444809 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444743 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.444809 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444757 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.445006 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.444987 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.445076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.445006 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.445076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.445016 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.445076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.445025 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.445076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.445034 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd284d9a-851f-48c8-a365-b456bffc3149-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.446376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.446355 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.446507 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.446442 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:31.446814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.446792 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.446918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.446790 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj" pod "bd284d9a-851f-48c8-a365-b456bffc3149" (UID: "bd284d9a-851f-48c8-a365-b456bffc3149"). InnerVolumeSpecName "kube-api-access-cstwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.546844 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.546810 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstwj\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-kube-api-access-cstwj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.547001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.546847 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd284d9a-851f-48c8-a365-b456bffc3149-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.547001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.546863 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd284d9a-851f-48c8-a365-b456bffc3149-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.547001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.546876 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd284d9a-851f-48c8-a365-b456bffc3149-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.607296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.607265 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6"] Oct 29 23:06:31.758023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.757997 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 23:06:31.796212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.796176 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp"] Oct 29 23:06:31.800746 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.800721 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-65hdp"] Oct 29 23:06:31.850180 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.850146 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume\") pod \"ade12afa-0df7-4590-858b-8e6bc8a9977a\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " Oct 29 23:06:31.850387 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.850281 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb\" (UniqueName: \"kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb\") pod \"ade12afa-0df7-4590-858b-8e6bc8a9977a\" (UID: \"ade12afa-0df7-4590-858b-8e6bc8a9977a\") " Oct 29 23:06:31.850641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.850613 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume" pod "ade12afa-0df7-4590-858b-8e6bc8a9977a" (UID: "ade12afa-0df7-4590-858b-8e6bc8a9977a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:31.850822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.850797 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade12afa-0df7-4590-858b-8e6bc8a9977a-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.852586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.852562 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb" pod "ade12afa-0df7-4590-858b-8e6bc8a9977a" (UID: "ade12afa-0df7-4590-858b-8e6bc8a9977a"). InnerVolumeSpecName "kube-api-access-k7qlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.952347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:31.952312 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qlb\" (UniqueName: \"kubernetes.io/projected/ade12afa-0df7-4590-858b-8e6bc8a9977a-kube-api-access-k7qlb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:32.431693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.431659 5367 generic.go:356] "Generic (PLEG): container finished" podID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerID="bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097" exitCode=0 Oct 29 23:06:32.431908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.431743 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" Oct 29 23:06:32.431908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.431749 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" event={"ID":"ade12afa-0df7-4590-858b-8e6bc8a9977a","Type":"ContainerDied","Data":"bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097"} Oct 29 23:06:32.431908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.431804 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb" event={"ID":"ade12afa-0df7-4590-858b-8e6bc8a9977a","Type":"ContainerDied","Data":"05e5952da19f75d999e6a793171d262b534c8640f995ceaae0a47690fa42c012"} Oct 29 23:06:32.431908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.431832 5367 scope.go:122] "RemoveContainer" containerID="bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097" Oct 29 23:06:32.443058 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.443041 5367 scope.go:122] "RemoveContainer" containerID="bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097" Oct 29 23:06:32.443309 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:32.443292 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097\": container with ID starting with bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097 not found: ID does not exist" containerID="bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097" Oct 29 23:06:32.443359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.443317 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097"} err="failed to get container status \"bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097\": rpc error: code = NotFound desc = could not find container \"bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097\": container with ID starting with bc4ddd807c3ef10dc2694556da42867476f50d3721172f0e5247bf929e04e097 not found: ID does not exist" Oct 29 23:06:32.488953 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.488923 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb"] Oct 29 23:06:32.494211 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.494184 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6bb06a-59ccd794fb-hxhxb"] Oct 29 23:06:32.828222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.828184 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:32.828616 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:32.828192 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:33.221681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:33.221606 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" path="/var/lib/kubelet/pods/ade12afa-0df7-4590-858b-8e6bc8a9977a/volumes" Oct 29 23:06:33.222134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:33.222119 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" path="/var/lib/kubelet/pods/bd284d9a-851f-48c8-a365-b456bffc3149/volumes" Oct 29 23:06:34.754287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.754261 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 23:06:34.886004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.885959 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886067 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886109 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886307 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886230 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886366 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886366 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886350 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886508 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886376 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886508 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886412 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886508 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886448 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token\") pod \"5be767f7-8da0-401b-8e43-caef5ec3fce7\" (UID: \"5be767f7-8da0-401b-8e43-caef5ec3fce7\") " Oct 29 23:06:34.886675 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886503 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:34.886780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886667 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:34.886849 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886814 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:34.886913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.886893 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:34.887034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.887015 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.887120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.887038 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.887190 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.887091 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be767f7-8da0-401b-8e43-caef5ec3fce7-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.887190 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.887168 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.887297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.887202 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:34.888401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.888377 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:34.888633 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.888608 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:34.888755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.888656 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:34.888805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.888774 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz" pod "5be767f7-8da0-401b-8e43-caef5ec3fce7" (UID: "5be767f7-8da0-401b-8e43-caef5ec3fce7"). InnerVolumeSpecName "kube-api-access-5wpzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:34.988479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.988446 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.988479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.988483 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wpzz\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-kube-api-access-5wpzz\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.988683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.988499 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.988683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.988513 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.988683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:34.988525 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be767f7-8da0-401b-8e43-caef5ec3fce7-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:35.495201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.495035 5367 generic.go:356] "Generic (PLEG): container finished" podID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerID="b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f" exitCode=0 Oct 29 23:06:35.495201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.495081 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" Oct 29 23:06:35.495201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.495118 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" event={"ID":"5be767f7-8da0-401b-8e43-caef5ec3fce7","Type":"ContainerDied","Data":"b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f"} Oct 29 23:06:35.495201 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.495178 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4" event={"ID":"5be767f7-8da0-401b-8e43-caef5ec3fce7","Type":"ContainerDied","Data":"e4feccd54f7a5fe2d0e80a96ad76f0360e783f4b07500df867be75e3e6310d0b"} Oct 29 23:06:35.495526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.495208 5367 scope.go:122] "RemoveContainer" containerID="b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f" Oct 29 23:06:35.508257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.508236 5367 scope.go:122] "RemoveContainer" containerID="b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f" Oct 29 23:06:35.508562 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:35.508538 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f\": container with ID starting with b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f not found: ID does not exist" containerID="b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f" Oct 29 23:06:35.508647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.508568 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f"} err="failed to get container status \"b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f\": rpc error: code = NotFound desc = could not find container \"b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f\": container with ID starting with b66e90a7f14abad5a0aa8ebe6d32a7b5406f6e6c26c8911be8be4605b0486a4f not found: ID does not exist" Oct 29 23:06:35.553266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.553237 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4"] Oct 29 23:06:35.566817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.566788 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-f7rq4"] Oct 29 23:06:35.616814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.616756 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj"] Oct 29 23:06:35.617081 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:35.617020 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" containerID="cri-o://2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867" gracePeriod=30 Oct 29 23:06:36.161394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161345 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n"] Oct 29 23:06:36.161925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161906 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" containerName="revoke-keys" Oct 29 23:06:36.161978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161927 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" containerName="revoke-keys" Oct 29 23:06:36.161978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161971 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8ad2e9e6-77df-4675-adc9-b5d0613786f9" containerName="revoke-keys" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161977 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad2e9e6-77df-4675-adc9-b5d0613786f9" containerName="revoke-keys" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161988 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.161993 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162000 5367 cpu_manager.go:414] "containerMap: removing container" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerName="payload-processing" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162005 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerName="payload-processing" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162022 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" Oct 29 23:06:36.162048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162026 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" Oct 29 23:06:36.162327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162138 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad2e9e6-77df-4675-adc9-b5d0613786f9" containerName="revoke-keys" Oct 29 23:06:36.162327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162149 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" containerName="istio-proxy" Oct 29 23:06:36.162327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162156 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade12afa-0df7-4590-858b-8e6bc8a9977a" containerName="payload-processing" Oct 29 23:06:36.162327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162165 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd284d9a-851f-48c8-a365-b456bffc3149" containerName="istio-proxy" Oct 29 23:06:36.162327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.162172 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" containerName="revoke-keys" Oct 29 23:06:36.166914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.166894 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.183104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.183075 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:36.184836 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.184807 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n"] Oct 29 23:06:36.194751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.194723 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:36.194887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.194797 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:06:36.194887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.194835 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:06:36.302609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.302573 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.303014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.302987 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lh8\" (UniqueName: \"kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.303121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.303073 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.404266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.404225 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lh8\" (UniqueName: \"kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.404477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.404280 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.404477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.404322 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.404950 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.404926 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.406811 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.406786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.425072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.425018 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lh8\" (UniqueName: \"kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.480168 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.480133 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:36.785693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:36.785661 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n"] Oct 29 23:06:36.786259 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:36.786232 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c660121_2410_47f1_a209_102d7b891ed0.slice/crio-5ce13c4c899a41c27088090c76647d5ec56f3db20f55c08a9f50c873850ab09b WatchSource:0}: Error finding container 5ce13c4c899a41c27088090c76647d5ec56f3db20f55c08a9f50c873850ab09b: Status 404 returned error can't find the container with id 5ce13c4c899a41c27088090c76647d5ec56f3db20f55c08a9f50c873850ab09b Oct 29 23:06:37.220973 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:37.220941 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be767f7-8da0-401b-8e43-caef5ec3fce7" path="/var/lib/kubelet/pods/5be767f7-8da0-401b-8e43-caef5ec3fce7/volumes" Oct 29 23:06:37.237683 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:37.237652 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8"] Oct 29 23:06:37.271266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:37.271235 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz"] Oct 29 23:06:37.515990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:37.515962 5367 generic.go:356] "Generic (PLEG): container finished" podID="8c660121-2410-47f1-a209-102d7b891ed0" containerID="89395d969042304c185b37cec66453788b74ff489af6acd8d97b0e0c857e0cad" exitCode=0 Oct 29 23:06:37.516145 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:37.516037 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" event={"ID":"8c660121-2410-47f1-a209-102d7b891ed0","Type":"ContainerDied","Data":"89395d969042304c185b37cec66453788b74ff489af6acd8d97b0e0c857e0cad"} Oct 29 23:06:37.516145 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:37.516090 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" event={"ID":"8c660121-2410-47f1-a209-102d7b891ed0","Type":"ContainerStarted","Data":"5ce13c4c899a41c27088090c76647d5ec56f3db20f55c08a9f50c873850ab09b"} Oct 29 23:06:38.386187 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.386153 5367 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:38.386588 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.386212 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:38.446606 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.446565 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg"] Oct 29 23:06:38.450787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.450763 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.474902 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.474868 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg"] Oct 29 23:06:38.629319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.629287 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.629516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.629373 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z65s\" (UniqueName: \"kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.629516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.629471 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.660277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.660254 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:38.731314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.731277 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.731484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.731382 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z65s\" (UniqueName: \"kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.731484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.731453 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.732020 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.731991 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.733847 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.733823 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.753140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.753108 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z65s\" (UniqueName: \"kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s\") pod \"maas-api-revoke-sub-c0ae1a01-z5wsg\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.766995 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.766973 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:38.819533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.819465 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln"] Oct 29 23:06:38.824319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.824289 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2ae845-m8cln"] Oct 29 23:06:38.839739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.832691 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca\") pod \"8c660121-2410-47f1-a209-102d7b891ed0\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " Oct 29 23:06:38.839739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.832816 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8\" (UniqueName: \"kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8\") pod \"8c660121-2410-47f1-a209-102d7b891ed0\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " Oct 29 23:06:38.839739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.832905 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca\") pod \"8c660121-2410-47f1-a209-102d7b891ed0\" (UID: \"8c660121-2410-47f1-a209-102d7b891ed0\") " Oct 29 23:06:38.839739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.833872 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca" pod "8c660121-2410-47f1-a209-102d7b891ed0" (UID: "8c660121-2410-47f1-a209-102d7b891ed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:38.840056 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.839998 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca" pod "8c660121-2410-47f1-a209-102d7b891ed0" (UID: "8c660121-2410-47f1-a209-102d7b891ed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:38.841815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.841793 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8" pod "8c660121-2410-47f1-a209-102d7b891ed0" (UID: "8c660121-2410-47f1-a209-102d7b891ed0"). InnerVolumeSpecName "kube-api-access-c2lh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:38.935993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.935718 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2lh8\" (UniqueName: \"kubernetes.io/projected/8c660121-2410-47f1-a209-102d7b891ed0-kube-api-access-c2lh8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.935993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.935775 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.935993 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:38.935825 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c660121-2410-47f1-a209-102d7b891ed0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:39.071354 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.071331 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg"] Oct 29 23:06:39.072682 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:39.072659 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4423cd5c_a24c_4a2f_a6c1_c08af04c852e.slice/crio-ff8b3586a9c4dae1bddf5165df6c9223671aa6b8e153d99649fed01ff812f184 WatchSource:0}: Error finding container ff8b3586a9c4dae1bddf5165df6c9223671aa6b8e153d99649fed01ff812f184: Status 404 returned error can't find the container with id ff8b3586a9c4dae1bddf5165df6c9223671aa6b8e153d99649fed01ff812f184 Oct 29 23:06:39.221551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.221469 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad2e9e6-77df-4675-adc9-b5d0613786f9" path="/var/lib/kubelet/pods/8ad2e9e6-77df-4675-adc9-b5d0613786f9/volumes" Oct 29 23:06:39.500025 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.499941 5367 scope.go:122] "RemoveContainer" containerID="35d55d84e75f10de7292bfcfc1cf6ec46446120cc0b7ceacf2c70f2c00c0dc20" Oct 29 23:06:39.510056 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.510034 5367 scope.go:122] "RemoveContainer" containerID="5d0711f218ed572dec339d12e0e74a448f093b7aecd824602723f9a291eead47" Oct 29 23:06:39.519415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.519396 5367 scope.go:122] "RemoveContainer" containerID="b1767dc328770d4b7edc7e3b50e2b1ff87a9e398d986688f6b8d879209381563" Oct 29 23:06:39.528519 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.528356 5367 scope.go:122] "RemoveContainer" containerID="0fcbb20bb8cc9bd16fe354219385a1a9ea7b2bc003f382e29d6c04c4114c2821" Oct 29 23:06:39.535817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.535798 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" Oct 29 23:06:39.535922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.535829 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n" event={"ID":"8c660121-2410-47f1-a209-102d7b891ed0","Type":"ContainerDied","Data":"5ce13c4c899a41c27088090c76647d5ec56f3db20f55c08a9f50c873850ab09b"} Oct 29 23:06:39.535922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.535883 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ce13c4c899a41c27088090c76647d5ec56f3db20f55c08a9f50c873850ab09b" Oct 29 23:06:39.538530 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.538508 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" event={"ID":"4423cd5c-a24c-4a2f-a6c1-c08af04c852e","Type":"ContainerStarted","Data":"6e87ddb79e7f8d9750052a70355a99489c44f4fef0c0b86a1d6ccd18695bfbf0"} Oct 29 23:06:39.538653 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.538552 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" event={"ID":"4423cd5c-a24c-4a2f-a6c1-c08af04c852e","Type":"ContainerStarted","Data":"ff8b3586a9c4dae1bddf5165df6c9223671aa6b8e153d99649fed01ff812f184"} Oct 29 23:06:39.539972 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.539954 5367 scope.go:122] "RemoveContainer" containerID="a7d0144c89412dba50fc3285b890125cafe788ee74a59bccb3eaa56505132e6d" Oct 29 23:06:39.604721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:39.604663 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" podStartSLOduration=1.604651331 podStartE2EDuration="1.604651331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:39.603297974 +0000 UTC m=+3066.497372898" watchObservedRunningTime="2026-10-29 23:06:39.604651331 +0000 UTC m=+3066.498726251" Oct 29 23:06:40.549185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:40.549149 5367 generic.go:356] "Generic (PLEG): container finished" podID="4423cd5c-a24c-4a2f-a6c1-c08af04c852e" containerID="6e87ddb79e7f8d9750052a70355a99489c44f4fef0c0b86a1d6ccd18695bfbf0" exitCode=0 Oct 29 23:06:40.549611 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:40.549228 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" event={"ID":"4423cd5c-a24c-4a2f-a6c1-c08af04c852e","Type":"ContainerDied","Data":"6e87ddb79e7f8d9750052a70355a99489c44f4fef0c0b86a1d6ccd18695bfbf0"} Oct 29 23:06:40.889866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:40.889842 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:06:41.062008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.061977 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062069 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062119 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062164 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062258 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062301 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062356 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062396 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062424 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert\") pod \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\" (UID: \"62d3306f-e6fa-4dba-97f5-c1b968c7ac19\") " Oct 29 23:06:41.062641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062557 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.062641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062605 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.062896 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062831 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.063073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.063041 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.063073 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.063066 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.063196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.063082 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.063196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.063044 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.063196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.062976 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:41.064408 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.064379 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:41.064534 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.064492 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.064855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.064837 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.064922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.064865 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r" pod "62d3306f-e6fa-4dba-97f5-c1b968c7ac19" (UID: "62d3306f-e6fa-4dba-97f5-c1b968c7ac19"). InnerVolumeSpecName "kube-api-access-bq94r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.164095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.164012 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.164095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.164043 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.164095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.164053 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.164095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.164062 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.164095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.164074 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq94r\" (UniqueName: \"kubernetes.io/projected/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-kube-api-access-bq94r\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.164095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.164086 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d3306f-e6fa-4dba-97f5-c1b968c7ac19-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.566610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.566570 5367 generic.go:356] "Generic (PLEG): container finished" podID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerID="2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867" exitCode=0 Oct 29 23:06:41.567155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.566737 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" Oct 29 23:06:41.567155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.566789 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" event={"ID":"62d3306f-e6fa-4dba-97f5-c1b968c7ac19","Type":"ContainerDied","Data":"2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867"} Oct 29 23:06:41.567155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.566852 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj" event={"ID":"62d3306f-e6fa-4dba-97f5-c1b968c7ac19","Type":"ContainerDied","Data":"42f3f5f8208a915c7757d6367b16507b19a1085f369b4f4704ada1b58b4a3c8f"} Oct 29 23:06:41.567155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.566882 5367 scope.go:122] "RemoveContainer" containerID="2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867" Oct 29 23:06:41.588554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.586246 5367 scope.go:122] "RemoveContainer" containerID="2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867" Oct 29 23:06:41.589009 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:41.588976 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867\": container with ID starting with 2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867 not found: ID does not exist" containerID="2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867" Oct 29 23:06:41.589986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.589017 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867"} err="failed to get container status \"2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867\": rpc error: code = NotFound desc = could not find container \"2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867\": container with ID starting with 2475b8380e3f3dd2c0cf226ef2834f48f7365339effa67c0e7bf6b93d0a81867 not found: ID does not exist" Oct 29 23:06:41.628733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.628670 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj"] Oct 29 23:06:41.641015 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.640978 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-rtqqj"] Oct 29 23:06:41.719252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.719227 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:41.770375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.770340 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca\") pod \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " Oct 29 23:06:41.770594 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.770451 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca\") pod \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " Oct 29 23:06:41.770594 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.770565 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s\" (UniqueName: \"kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s\") pod \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\" (UID: \"4423cd5c-a24c-4a2f-a6c1-c08af04c852e\") " Oct 29 23:06:41.770747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.770721 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca" pod "4423cd5c-a24c-4a2f-a6c1-c08af04c852e" (UID: "4423cd5c-a24c-4a2f-a6c1-c08af04c852e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:41.770987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.770961 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.772610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.772585 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s" pod "4423cd5c-a24c-4a2f-a6c1-c08af04c852e" (UID: "4423cd5c-a24c-4a2f-a6c1-c08af04c852e"). InnerVolumeSpecName "kube-api-access-7z65s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.772756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.772673 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca" pod "4423cd5c-a24c-4a2f-a6c1-c08af04c852e" (UID: "4423cd5c-a24c-4a2f-a6c1-c08af04c852e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:41.871797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.871712 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z65s\" (UniqueName: \"kubernetes.io/projected/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-kube-api-access-7z65s\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.871797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:41.871753 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4423cd5c-a24c-4a2f-a6c1-c08af04c852e-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:42.238027 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.237920 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerName="payload-processing" containerID="cri-o://58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a" gracePeriod=25 Oct 29 23:06:42.272515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.272469 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerName="payload-pre-processing" containerID="cri-o://bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24" gracePeriod=25 Oct 29 23:06:42.575121 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.575091 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" Oct 29 23:06:42.575562 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.575122 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg" event={"ID":"4423cd5c-a24c-4a2f-a6c1-c08af04c852e","Type":"ContainerDied","Data":"ff8b3586a9c4dae1bddf5165df6c9223671aa6b8e153d99649fed01ff812f184"} Oct 29 23:06:42.575562 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.575178 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8b3586a9c4dae1bddf5165df6c9223671aa6b8e153d99649fed01ff812f184" Oct 29 23:06:42.827376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.827297 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:42.827526 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:42.827296 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:43.226129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.226100 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" path="/var/lib/kubelet/pods/62d3306f-e6fa-4dba-97f5-c1b968c7ac19/volumes" Oct 29 23:06:43.462051 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.462026 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 23:06:43.480467 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.480445 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 23:06:43.484008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.483980 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq\" (UniqueName: \"kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq\") pod \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " Oct 29 23:06:43.484120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484060 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume\") pod \"5aa8264e-4f5e-4980-8cbb-8863df25b955\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " Oct 29 23:06:43.484120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484100 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume\") pod \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\" (UID: \"786ff7e6-15b5-4716-b6b5-ac7f306d3a14\") " Oct 29 23:06:43.484239 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484223 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj\" (UniqueName: \"kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj\") pod \"5aa8264e-4f5e-4980-8cbb-8863df25b955\" (UID: \"5aa8264e-4f5e-4980-8cbb-8863df25b955\") " Oct 29 23:06:43.484414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484391 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume" pod "5aa8264e-4f5e-4980-8cbb-8863df25b955" (UID: "5aa8264e-4f5e-4980-8cbb-8863df25b955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:43.484498 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484486 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume" pod "786ff7e6-15b5-4716-b6b5-ac7f306d3a14" (UID: "786ff7e6-15b5-4716-b6b5-ac7f306d3a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:43.484581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484566 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa8264e-4f5e-4980-8cbb-8863df25b955-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:43.484644 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.484583 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:43.486042 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.486012 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq" pod "786ff7e6-15b5-4716-b6b5-ac7f306d3a14" (UID: "786ff7e6-15b5-4716-b6b5-ac7f306d3a14"). InnerVolumeSpecName "kube-api-access-84tzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:43.486130 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.486073 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj" pod "5aa8264e-4f5e-4980-8cbb-8863df25b955" (UID: "5aa8264e-4f5e-4980-8cbb-8863df25b955"). InnerVolumeSpecName "kube-api-access-btsgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:43.584843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.584807 5367 generic.go:356] "Generic (PLEG): container finished" podID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerID="6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1" exitCode=0 Oct 29 23:06:43.584843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.584852 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" Oct 29 23:06:43.585374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.584889 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" event={"ID":"5aa8264e-4f5e-4980-8cbb-8863df25b955","Type":"ContainerDied","Data":"6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1"} Oct 29 23:06:43.585374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.584939 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887" event={"ID":"5aa8264e-4f5e-4980-8cbb-8863df25b955","Type":"ContainerDied","Data":"1fdc3b6bc4cfde8078cc78ed4bc8cb48e638e3dea98cbd5e088d8bfa8908f7e5"} Oct 29 23:06:43.585374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.584965 5367 scope.go:122] "RemoveContainer" containerID="6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1" Oct 29 23:06:43.585374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.585368 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsgj\" (UniqueName: \"kubernetes.io/projected/5aa8264e-4f5e-4980-8cbb-8863df25b955-kube-api-access-btsgj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:43.585538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.585385 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84tzq\" (UniqueName: \"kubernetes.io/projected/786ff7e6-15b5-4716-b6b5-ac7f306d3a14-kube-api-access-84tzq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:43.586429 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.586407 5367 generic.go:356] "Generic (PLEG): container finished" podID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerID="7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51" exitCode=0 Oct 29 23:06:43.586549 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.586462 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" Oct 29 23:06:43.586549 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.586506 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" event={"ID":"786ff7e6-15b5-4716-b6b5-ac7f306d3a14","Type":"ContainerDied","Data":"7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51"} Oct 29 23:06:43.586634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.586554 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj" event={"ID":"786ff7e6-15b5-4716-b6b5-ac7f306d3a14","Type":"ContainerDied","Data":"981cdd658c6d4a8b35c1aeb03fe97d37f17326c03ea3f5a525251ffe749a445a"} Oct 29 23:06:43.595119 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.595097 5367 scope.go:122] "RemoveContainer" containerID="6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1" Oct 29 23:06:43.595430 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:43.595370 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1\": container with ID starting with 6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1 not found: ID does not exist" containerID="6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1" Oct 29 23:06:43.595430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.595394 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1"} err="failed to get container status \"6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1\": rpc error: code = NotFound desc = could not find container \"6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1\": container with ID starting with 6fa508a37e216c2073ce1f94819b343f0807e2324e161d33113fb1344f5889b1 not found: ID does not exist" Oct 29 23:06:43.595430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.595411 5367 scope.go:122] "RemoveContainer" containerID="7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51" Oct 29 23:06:43.604728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.604708 5367 scope.go:122] "RemoveContainer" containerID="7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51" Oct 29 23:06:43.604970 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:43.604952 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51\": container with ID starting with 7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51 not found: ID does not exist" containerID="7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51" Oct 29 23:06:43.605028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.604976 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51"} err="failed to get container status \"7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51\": rpc error: code = NotFound desc = could not find container \"7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51\": container with ID starting with 7c8e4ca4829bafb5effe158d12972191f4e0e3ed791ee7266a922ef74c364e51 not found: ID does not exist" Oct 29 23:06:43.641060 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.641031 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887"] Oct 29 23:06:43.644065 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.644042 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-36b46d-754ddff76c-kj887"] Oct 29 23:06:43.680660 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.680577 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj"] Oct 29 23:06:43.684087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:43.684060 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36b46d-78c99bbd9b-fz2mj"] Oct 29 23:06:45.221037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:45.221007 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" path="/var/lib/kubelet/pods/5aa8264e-4f5e-4980-8cbb-8863df25b955/volumes" Oct 29 23:06:45.221423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:45.221393 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" path="/var/lib/kubelet/pods/786ff7e6-15b5-4716-b6b5-ac7f306d3a14/volumes" Oct 29 23:06:45.440436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:45.438681 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq"] Oct 29 23:06:45.440436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:45.438967 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" containerID="cri-o://bea0bb37ff5ba9be8a31851962ca8d388fcd17075a5a7868a56ef41eaaa72a8c" gracePeriod=30 Oct 29 23:06:46.158555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.158522 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp"] Oct 29 23:06:46.159016 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159000 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerName="payload-pre-processing" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159017 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerName="payload-pre-processing" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159034 5367 cpu_manager.go:414] "containerMap: removing container" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerName="payload-processing" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159040 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerName="payload-processing" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159058 5367 cpu_manager.go:414] "containerMap: removing container" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159063 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159090 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8c660121-2410-47f1-a209-102d7b891ed0" containerName="revoke-keys" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159094 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c660121-2410-47f1-a209-102d7b891ed0" containerName="revoke-keys" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159101 5367 cpu_manager.go:414] "containerMap: removing container" podUID="4423cd5c-a24c-4a2f-a6c1-c08af04c852e" containerName="revoke-keys" Oct 29 23:06:46.159105 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159106 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4423cd5c-a24c-4a2f-a6c1-c08af04c852e" containerName="revoke-keys" Oct 29 23:06:46.159396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159170 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d3306f-e6fa-4dba-97f5-c1b968c7ac19" containerName="istio-proxy" Oct 29 23:06:46.159396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159181 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c660121-2410-47f1-a209-102d7b891ed0" containerName="revoke-keys" Oct 29 23:06:46.159396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159186 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="4423cd5c-a24c-4a2f-a6c1-c08af04c852e" containerName="revoke-keys" Oct 29 23:06:46.159396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159192 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa8264e-4f5e-4980-8cbb-8863df25b955" containerName="payload-pre-processing" Oct 29 23:06:46.159396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.159199 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="786ff7e6-15b5-4716-b6b5-ac7f306d3a14" containerName="payload-processing" Oct 29 23:06:46.163887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.163871 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.182673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.182649 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp"] Oct 29 23:06:46.188136 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.188108 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:46.192829 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.192805 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:46.211557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.211534 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.211684 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.211564 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg9jp\" (UniqueName: \"kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.211778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.211759 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.224484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.224465 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:06:46.224826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.224544 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:06:46.313113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.313082 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.313292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.313184 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.313292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.313212 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg9jp\" (UniqueName: \"kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.313846 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.313823 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.315522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.315501 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.334219 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.334192 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg9jp\" (UniqueName: \"kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.476605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.476516 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:46.567539 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.567394 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz"] Oct 29 23:06:46.606114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.606064 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7"] Oct 29 23:06:46.628531 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.628493 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" containerID="cri-o://428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4" gracePeriod=45 Oct 29 23:06:46.787046 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.786971 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp"] Oct 29 23:06:46.976803 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:46.976771 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 23:06:47.029943 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.029860 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache\") pod \"d926f8d9-c648-4312-9df4-488ffa2b300a\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " Oct 29 23:06:47.030112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.029943 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm\") pod \"d926f8d9-c648-4312-9df4-488ffa2b300a\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " Oct 29 23:06:47.030112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.030031 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs\") pod \"d926f8d9-c648-4312-9df4-488ffa2b300a\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " Oct 29 23:06:47.030112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.030070 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home\") pod \"d926f8d9-c648-4312-9df4-488ffa2b300a\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " Oct 29 23:06:47.030302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.030147 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk\" (UniqueName: \"kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk\") pod \"d926f8d9-c648-4312-9df4-488ffa2b300a\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " Oct 29 23:06:47.030302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.030198 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir\") pod \"d926f8d9-c648-4312-9df4-488ffa2b300a\" (UID: \"d926f8d9-c648-4312-9df4-488ffa2b300a\") " Oct 29 23:06:47.031235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.031203 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir" pod "d926f8d9-c648-4312-9df4-488ffa2b300a" (UID: "d926f8d9-c648-4312-9df4-488ffa2b300a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.031640 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.031595 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache" pod "d926f8d9-c648-4312-9df4-488ffa2b300a" (UID: "d926f8d9-c648-4312-9df4-488ffa2b300a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.032299 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.032267 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home" pod "d926f8d9-c648-4312-9df4-488ffa2b300a" (UID: "d926f8d9-c648-4312-9df4-488ffa2b300a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.035252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.035225 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs" pod "d926f8d9-c648-4312-9df4-488ffa2b300a" (UID: "d926f8d9-c648-4312-9df4-488ffa2b300a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:47.039783 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.039750 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk" pod "d926f8d9-c648-4312-9df4-488ffa2b300a" (UID: "d926f8d9-c648-4312-9df4-488ffa2b300a"). InnerVolumeSpecName "kube-api-access-ptxxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:47.039893 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.039818 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm" pod "d926f8d9-c648-4312-9df4-488ffa2b300a" (UID: "d926f8d9-c648-4312-9df4-488ffa2b300a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.131883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.131853 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d926f8d9-c648-4312-9df4-488ffa2b300a-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.131883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.131880 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.131883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.131890 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxxk\" (UniqueName: \"kubernetes.io/projected/d926f8d9-c648-4312-9df4-488ffa2b300a-kube-api-access-ptxxk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.132124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.131899 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.132124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.131908 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.132124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.131915 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d926f8d9-c648-4312-9df4-488ffa2b300a-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.625133 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.625038 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" event={"ID":"e3f13d0b-1743-493f-abed-b311f49f3ea0","Type":"ContainerStarted","Data":"e6d38e70c2e1a0a28b28ab6f508030e2341931be2b1c55c9a705ca06612b6525"} Oct 29 23:06:47.625133 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.625105 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" event={"ID":"e3f13d0b-1743-493f-abed-b311f49f3ea0","Type":"ContainerStarted","Data":"e2cd0569f59cc3534acfe6198fe1a7f313ead60052ad1576cdd3220306cfb33b"} Oct 29 23:06:47.626890 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.626864 5367 generic.go:356] "Generic (PLEG): container finished" podID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerID="428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4" exitCode=0 Oct 29 23:06:47.627026 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.626918 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" Oct 29 23:06:47.627026 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.626917 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" event={"ID":"d926f8d9-c648-4312-9df4-488ffa2b300a","Type":"ContainerDied","Data":"428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4"} Oct 29 23:06:47.627104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.627031 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" event={"ID":"d926f8d9-c648-4312-9df4-488ffa2b300a","Type":"ContainerDied","Data":"5b265ac408634ea6df33b38388f22ece454f4c5ad6b467c2382a9ad7890885a5"} Oct 29 23:06:47.627104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.627051 5367 scope.go:122] "RemoveContainer" containerID="428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4" Oct 29 23:06:47.638444 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.638407 5367 scope.go:122] "RemoveContainer" containerID="428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4" Oct 29 23:06:47.638832 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:47.638806 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4\": container with ID starting with 428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4 not found: ID does not exist" containerID="428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4" Oct 29 23:06:47.638908 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.638839 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4"} err="failed to get container status \"428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4\": rpc error: code = NotFound desc = could not find container \"428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4\": container with ID starting with 428effbb8239e179c0f1279405efc83fb54f87554da6f1aa527a37ca23ff6da4 not found: ID does not exist" Oct 29 23:06:47.669212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.669161 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" podStartSLOduration=1.6691457650000001 podStartE2EDuration="1.669145765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:47.667857422 +0000 UTC m=+3074.561932342" watchObservedRunningTime="2026-10-29 23:06:47.669145765 +0000 UTC m=+3074.563220684" Oct 29 23:06:47.725047 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.725011 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6"] Oct 29 23:06:47.730088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.730054 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6"] Oct 29 23:06:47.831725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:47.831667 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-2-w6-92fd86-kserve-89d548559-4s9l6" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:06:48.636963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:48.636928 5367 generic.go:356] "Generic (PLEG): container finished" podID="e3f13d0b-1743-493f-abed-b311f49f3ea0" containerID="e6d38e70c2e1a0a28b28ab6f508030e2341931be2b1c55c9a705ca06612b6525" exitCode=0 Oct 29 23:06:48.637396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:48.637013 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" event={"ID":"e3f13d0b-1743-493f-abed-b311f49f3ea0","Type":"ContainerDied","Data":"e6d38e70c2e1a0a28b28ab6f508030e2341931be2b1c55c9a705ca06612b6525"} Oct 29 23:06:48.741440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:48.741406 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n"] Oct 29 23:06:48.744154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:48.744124 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8f1627-df69n"] Oct 29 23:06:49.204884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.204848 5367 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:49.205049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.204904 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:49.220845 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.220816 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c660121-2410-47f1-a209-102d7b891ed0" path="/var/lib/kubelet/pods/8c660121-2410-47f1-a209-102d7b891ed0/volumes" Oct 29 23:06:49.221279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.221260 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" path="/var/lib/kubelet/pods/d926f8d9-c648-4312-9df4-488ffa2b300a/volumes" Oct 29 23:06:49.788888 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.788854 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:49.862715 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.862675 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca\") pod \"e3f13d0b-1743-493f-abed-b311f49f3ea0\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " Oct 29 23:06:49.862924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.862802 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca\") pod \"e3f13d0b-1743-493f-abed-b311f49f3ea0\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " Oct 29 23:06:49.862924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.862838 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp\" (UniqueName: \"kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp\") pod \"e3f13d0b-1743-493f-abed-b311f49f3ea0\" (UID: \"e3f13d0b-1743-493f-abed-b311f49f3ea0\") " Oct 29 23:06:49.863286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.863253 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca" pod "e3f13d0b-1743-493f-abed-b311f49f3ea0" (UID: "e3f13d0b-1743-493f-abed-b311f49f3ea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:49.864926 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.864893 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca" pod "e3f13d0b-1743-493f-abed-b311f49f3ea0" (UID: "e3f13d0b-1743-493f-abed-b311f49f3ea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:49.865041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.865022 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp" pod "e3f13d0b-1743-493f-abed-b311f49f3ea0" (UID: "e3f13d0b-1743-493f-abed-b311f49f3ea0"). InnerVolumeSpecName "kube-api-access-bg9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:49.964115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.964034 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg9jp\" (UniqueName: \"kubernetes.io/projected/e3f13d0b-1743-493f-abed-b311f49f3ea0-kube-api-access-bg9jp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:49.964115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.964069 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:49.964115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:49.964084 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f13d0b-1743-493f-abed-b311f49f3ea0-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.535981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.535941 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:50.536955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.536925 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:50.621664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.621631 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k"] Oct 29 23:06:50.635738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.635676 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-92fd86-rsc7k"] Oct 29 23:06:50.654511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.654464 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" Oct 29 23:06:50.654511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.654495 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp" event={"ID":"e3f13d0b-1743-493f-abed-b311f49f3ea0","Type":"ContainerDied","Data":"e2cd0569f59cc3534acfe6198fe1a7f313ead60052ad1576cdd3220306cfb33b"} Oct 29 23:06:50.654740 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.654548 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2cd0569f59cc3534acfe6198fe1a7f313ead60052ad1576cdd3220306cfb33b" Oct 29 23:06:50.673192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.672213 5367 generic.go:356] "Generic (PLEG): container finished" podID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerID="bea0bb37ff5ba9be8a31851962ca8d388fcd17075a5a7868a56ef41eaaa72a8c" exitCode=0 Oct 29 23:06:50.673192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.672305 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" event={"ID":"c28b49d4-923a-418f-96dc-2cca550cc1e5","Type":"ContainerDied","Data":"bea0bb37ff5ba9be8a31851962ca8d388fcd17075a5a7868a56ef41eaaa72a8c"} Oct 29 23:06:50.673192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.672404 5367 scope.go:122] "RemoveContainer" containerID="b3b47116b2b94e1c6b7c88fb65b667e4162b88c5bbda5ef9e69839cd5bc34ae4" Oct 29 23:06:50.709182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.709125 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 23:06:50.780832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.780797 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.780981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.780850 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.780981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.780877 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.780981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.780907 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.780981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.780940 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.780981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.780975 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.781273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781045 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.781273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781075 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.781273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781140 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert\") pod \"c28b49d4-923a-418f-96dc-2cca550cc1e5\" (UID: \"c28b49d4-923a-418f-96dc-2cca550cc1e5\") " Oct 29 23:06:50.781627 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781133 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.781806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781245 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.781920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781899 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:50.782001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781428 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.782001 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.781595 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.783596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.783560 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.783752 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.783728 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.783841 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.783824 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "kube-api-access-4lq5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.783946 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.783926 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo" pod "c28b49d4-923a-418f-96dc-2cca550cc1e5" (UID: "c28b49d4-923a-418f-96dc-2cca550cc1e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:50.883174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883140 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883169 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883179 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883189 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883198 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lq5g\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-kube-api-access-4lq5g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883207 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883215 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883223 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28b49d4-923a-418f-96dc-2cca550cc1e5-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.883623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.883230 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28b49d4-923a-418f-96dc-2cca550cc1e5-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.985085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:50.985005 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx"] Oct 29 23:06:51.221446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.221414 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0572a5-4a37-46d7-8229-5ac6a32a0fcd" path="/var/lib/kubelet/pods/2d0572a5-4a37-46d7-8229-5ac6a32a0fcd/volumes" Oct 29 23:06:51.568711 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.568667 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerName="payload-processing" containerID="cri-o://8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09" gracePeriod=25 Oct 29 23:06:51.606804 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.606759 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" containerID="cri-o://a41ba8297796640ed4535cfafab76591189f2cd8f7f2aa2e646769906f1a3ccc" gracePeriod=25 Oct 29 23:06:51.681318 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.681287 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" Oct 29 23:06:51.681481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.681322 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq" event={"ID":"c28b49d4-923a-418f-96dc-2cca550cc1e5","Type":"ContainerDied","Data":"f63d3eee81980e7ad4270fa0a1d313e8ed835ca4967a382055c2bb6ca4deb5ff"} Oct 29 23:06:51.681481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.681381 5367 scope.go:122] "RemoveContainer" containerID="bea0bb37ff5ba9be8a31851962ca8d388fcd17075a5a7868a56ef41eaaa72a8c" Oct 29 23:06:51.754996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.754958 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq"] Oct 29 23:06:51.768806 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:51.768774 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-p42lq"] Oct 29 23:06:52.330101 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:06:52.330064 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:06:53.220971 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:53.220940 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" path="/var/lib/kubelet/pods/c28b49d4-923a-418f-96dc-2cca550cc1e5/volumes" Oct 29 23:06:55.729911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:55.729880 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5"] Oct 29 23:06:55.730439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:55.730122 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" containerID="cri-o://316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74" gracePeriod=30 Oct 29 23:06:56.374161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374132 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp"] Oct 29 23:06:56.374626 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374611 5367 cpu_manager.go:414] "containerMap: removing container" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374627 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374641 5367 cpu_manager.go:414] "containerMap: removing container" podUID="e3f13d0b-1743-493f-abed-b311f49f3ea0" containerName="revoke-keys" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374645 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f13d0b-1743-493f-abed-b311f49f3ea0" containerName="revoke-keys" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374663 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374668 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374675 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" Oct 29 23:06:56.374710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374680 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" Oct 29 23:06:56.375158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374754 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" Oct 29 23:06:56.375158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374768 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="d926f8d9-c648-4312-9df4-488ffa2b300a" containerName="main" Oct 29 23:06:56.375158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.374777 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f13d0b-1743-493f-abed-b311f49f3ea0" containerName="revoke-keys" Oct 29 23:06:56.379692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.379675 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.398114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.398087 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp"] Oct 29 23:06:56.408055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.408031 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:06:56.408237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.408078 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:06:56.408237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.408092 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:56.408374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.408120 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:56.440235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.440198 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5ml\" (UniqueName: \"kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.440457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.440270 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.440457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.440432 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.541203 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.541174 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.541375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.541317 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5ml\" (UniqueName: \"kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.541375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.541368 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.543035 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.541904 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.547074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.547042 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.562428 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.562390 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5ml\" (UniqueName: \"kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml\") pod \"maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.692579 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.692495 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:56.982161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.982086 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9"] Oct 29 23:06:56.993758 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:56.993732 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp"] Oct 29 23:06:56.995144 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:06:56.995119 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf06c62_1105_4ca9_91ce_493e15b0cfa5.slice/crio-263c04195b199039adbdd94eec9e6312ebfc8c8afe8b37bec085a112a85139b8 WatchSource:0}: Error finding container 263c04195b199039adbdd94eec9e6312ebfc8c8afe8b37bec085a112a85139b8: Status 404 returned error can't find the container with id 263c04195b199039adbdd94eec9e6312ebfc8c8afe8b37bec085a112a85139b8 Oct 29 23:06:57.733400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:57.733363 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" event={"ID":"daf06c62-1105-4ca9-91ce-493e15b0cfa5","Type":"ContainerStarted","Data":"1fefc68eb64543eced577322d94d3148092f29b0e5e603936983af420bcf1994"} Oct 29 23:06:57.733591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:57.733427 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" event={"ID":"daf06c62-1105-4ca9-91ce-493e15b0cfa5","Type":"ContainerStarted","Data":"263c04195b199039adbdd94eec9e6312ebfc8c8afe8b37bec085a112a85139b8"} Oct 29 23:06:57.777983 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:57.777927 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" podStartSLOduration=1.777913469 podStartE2EDuration="1.777913469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:57.775126946 +0000 UTC m=+3084.669201868" watchObservedRunningTime="2026-10-29 23:06:57.777913469 +0000 UTC m=+3084.671988390" Oct 29 23:06:58.744181 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:58.744148 5367 generic.go:356] "Generic (PLEG): container finished" podID="daf06c62-1105-4ca9-91ce-493e15b0cfa5" containerID="1fefc68eb64543eced577322d94d3148092f29b0e5e603936983af420bcf1994" exitCode=0 Oct 29 23:06:58.744587 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:58.744211 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" event={"ID":"daf06c62-1105-4ca9-91ce-493e15b0cfa5","Type":"ContainerDied","Data":"1fefc68eb64543eced577322d94d3148092f29b0e5e603936983af420bcf1994"} Oct 29 23:06:59.489586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.489545 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:59.489791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.489555 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:59.889199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.889173 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:06:59.978889 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.978858 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca\") pod \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " Oct 29 23:06:59.979064 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.978993 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml\" (UniqueName: \"kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml\") pod \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " Oct 29 23:06:59.979064 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.979024 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca\") pod \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\" (UID: \"daf06c62-1105-4ca9-91ce-493e15b0cfa5\") " Oct 29 23:06:59.979277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.979248 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca" pod "daf06c62-1105-4ca9-91ce-493e15b0cfa5" (UID: "daf06c62-1105-4ca9-91ce-493e15b0cfa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:59.980990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.980969 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca" pod "daf06c62-1105-4ca9-91ce-493e15b0cfa5" (UID: "daf06c62-1105-4ca9-91ce-493e15b0cfa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:59.981216 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:06:59.981198 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml" pod "daf06c62-1105-4ca9-91ce-493e15b0cfa5" (UID: "daf06c62-1105-4ca9-91ce-493e15b0cfa5"). InnerVolumeSpecName "kube-api-access-fw5ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.022760 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.022672 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9"] Oct 29 23:07:00.023148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.023133 5367 cpu_manager.go:414] "containerMap: removing container" podUID="daf06c62-1105-4ca9-91ce-493e15b0cfa5" containerName="revoke-keys" Oct 29 23:07:00.023196 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.023151 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf06c62-1105-4ca9-91ce-493e15b0cfa5" containerName="revoke-keys" Oct 29 23:07:00.023278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.023268 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf06c62-1105-4ca9-91ce-493e15b0cfa5" containerName="revoke-keys" Oct 29 23:07:00.023311 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.023280 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28b49d4-923a-418f-96dc-2cca550cc1e5" containerName="istio-proxy" Oct 29 23:07:00.026145 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.026130 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.040778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.040749 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd"] Oct 29 23:07:00.045537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.045511 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.046861 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.046838 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9"] Oct 29 23:07:00.065996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.065965 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd"] Oct 29 23:07:00.080641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.080591 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.080641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.080641 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.080931 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.080765 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xkr\" (UniqueName: \"kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.080931 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.080859 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5ml\" (UniqueName: \"kubernetes.io/projected/daf06c62-1105-4ca9-91ce-493e15b0cfa5-kube-api-access-fw5ml\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.080931 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.080870 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.080931 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.080881 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf06c62-1105-4ca9-91ce-493e15b0cfa5-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.181915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.181874 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.182087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.182034 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.182087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.182075 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.182238 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.182099 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwqw\" (UniqueName: \"kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.182291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.182256 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xkr\" (UniqueName: \"kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.182427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.182395 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.182916 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.182892 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.185070 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.185046 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.203132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.203103 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xkr\" (UniqueName: \"kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr\") pod \"maas-api-revoke-sub-6be4472b-9qzj9\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.283531 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.283448 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.283739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.283546 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwqw\" (UniqueName: \"kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.283739 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.283620 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.284163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.284141 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.285953 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.285931 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.304404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.304382 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwqw\" (UniqueName: \"kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw\") pod \"maas-api-revoke-sub-cd36dee8-b7jjd\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.341067 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.341043 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:00.362216 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.362195 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:00.536212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.536113 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:00.536941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.536911 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:00.646410 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:07:00.646375 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5589ef46_bc4b_4e5f_a94e_369414e4d230.slice/crio-8cebaf11f47dc69b99bc2f1399cf004d85b2610bda833a1d73867e8c24bd1cca WatchSource:0}: Error finding container 8cebaf11f47dc69b99bc2f1399cf004d85b2610bda833a1d73867e8c24bd1cca: Status 404 returned error can't find the container with id 8cebaf11f47dc69b99bc2f1399cf004d85b2610bda833a1d73867e8c24bd1cca Oct 29 23:07:00.646410 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.646406 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9"] Oct 29 23:07:00.684895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.684867 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd"] Oct 29 23:07:00.685966 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:07:00.685941 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc181c1f2_c4dc_4687_8f3d_621e0da99b7d.slice/crio-ad3b29c8bc0fc27c4ef389c450fe1827e007313e8c400a3c7babf845443e3905 WatchSource:0}: Error finding container ad3b29c8bc0fc27c4ef389c450fe1827e007313e8c400a3c7babf845443e3905: Status 404 returned error can't find the container with id ad3b29c8bc0fc27c4ef389c450fe1827e007313e8c400a3c7babf845443e3905 Oct 29 23:07:00.760861 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.760828 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" event={"ID":"5589ef46-bc4b-4e5f-a94e-369414e4d230","Type":"ContainerStarted","Data":"8cebaf11f47dc69b99bc2f1399cf004d85b2610bda833a1d73867e8c24bd1cca"} Oct 29 23:07:00.762078 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.762050 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" event={"ID":"c181c1f2-c4dc-4687-8f3d-621e0da99b7d","Type":"ContainerStarted","Data":"ad3b29c8bc0fc27c4ef389c450fe1827e007313e8c400a3c7babf845443e3905"} Oct 29 23:07:00.763370 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.763352 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" Oct 29 23:07:00.763509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.763385 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp" event={"ID":"daf06c62-1105-4ca9-91ce-493e15b0cfa5","Type":"ContainerDied","Data":"263c04195b199039adbdd94eec9e6312ebfc8c8afe8b37bec085a112a85139b8"} Oct 29 23:07:00.763509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:00.763424 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="263c04195b199039adbdd94eec9e6312ebfc8c8afe8b37bec085a112a85139b8" Oct 29 23:07:01.081996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.081966 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:07:01.182635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.182608 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq"] Oct 29 23:07:01.193303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193271 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193353 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193389 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193423 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193490 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193530 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193583 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193587 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.193978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193634 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket\") pod \"278233b5-18e7-44bf-940f-76c1e6ee8c42\" (UID: \"278233b5-18e7-44bf-940f-76c1e6ee8c42\") " Oct 29 23:07:01.193978 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.193870 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.194726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.194265 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.194726 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.194472 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:01.194893 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.194775 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.195126 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.195101 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.195220 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.194794 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.195220 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.195207 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.195344 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.195224 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278233b5-18e7-44bf-940f-76c1e6ee8c42-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.196389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.196366 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "kube-api-access-gjn6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.196723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.196672 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.196836 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.196817 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.196937 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.196920 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo" pod "278233b5-18e7-44bf-940f-76c1e6ee8c42" (UID: "278233b5-18e7-44bf-940f-76c1e6ee8c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:01.209999 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.209979 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9"] Oct 29 23:07:01.296371 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.296328 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.296371 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.296360 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjn6g\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-kube-api-access-gjn6g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.296371 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.296377 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.296628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.296392 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278233b5-18e7-44bf-940f-76c1e6ee8c42-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.296628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.296404 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278233b5-18e7-44bf-940f-76c1e6ee8c42-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.772298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.772266 5367 generic.go:356] "Generic (PLEG): container finished" podID="c181c1f2-c4dc-4687-8f3d-621e0da99b7d" containerID="07891d5ef54b00821eec34641766c736cb539228ac5186b7fffa87b9f224bdad" exitCode=0 Oct 29 23:07:01.772438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.772344 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" event={"ID":"c181c1f2-c4dc-4687-8f3d-621e0da99b7d","Type":"ContainerDied","Data":"07891d5ef54b00821eec34641766c736cb539228ac5186b7fffa87b9f224bdad"} Oct 29 23:07:01.774055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.774030 5367 generic.go:356] "Generic (PLEG): container finished" podID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerID="316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74" exitCode=0 Oct 29 23:07:01.774163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.774070 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" Oct 29 23:07:01.774163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.774105 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" event={"ID":"278233b5-18e7-44bf-940f-76c1e6ee8c42","Type":"ContainerDied","Data":"316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74"} Oct 29 23:07:01.774282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.774159 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5" event={"ID":"278233b5-18e7-44bf-940f-76c1e6ee8c42","Type":"ContainerDied","Data":"b3920382562bb4a74fed48e5bb703b11acaa83fd3d16c6c1df6d0214c353d3c7"} Oct 29 23:07:01.774282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.774190 5367 scope.go:122] "RemoveContainer" containerID="316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74" Oct 29 23:07:01.776019 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.775985 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" event={"ID":"5589ef46-bc4b-4e5f-a94e-369414e4d230","Type":"ContainerStarted","Data":"fcfee064f9c9179972185575e9b08f838860d7c0796d15557262f0bee2606b9f"} Oct 29 23:07:01.832468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.832442 5367 scope.go:122] "RemoveContainer" containerID="316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74" Oct 29 23:07:01.832792 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:01.832770 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74\": container with ID starting with 316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74 not found: ID does not exist" containerID="316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74" Oct 29 23:07:01.832837 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.832803 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74"} err="failed to get container status \"316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74\": rpc error: code = NotFound desc = could not find container \"316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74\": container with ID starting with 316f39d4cabe9e090c92116c7964fb03e7c419a4cd3d79efe8f2805b07e41b74 not found: ID does not exist" Oct 29 23:07:01.858959 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.858905 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" podStartSLOduration=2.858892451 podStartE2EDuration="2.858892451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:01.857383967 +0000 UTC m=+3088.751458889" watchObservedRunningTime="2026-10-29 23:07:01.858892451 +0000 UTC m=+3088.752967372" Oct 29 23:07:01.919744 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.919713 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5"] Oct 29 23:07:01.933022 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.932980 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-z48b5"] Oct 29 23:07:01.983394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:01.983355 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" podUID="c8558681-9856-4931-a120-1008a32b6300" containerName="payload-processing" containerID="cri-o://b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e" gracePeriod=25 Oct 29 23:07:02.525988 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.525959 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 23:07:02.549224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.549200 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 23:07:02.609729 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.609681 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume\") pod \"44ab836b-2090-498b-99b5-34f1256aa9b1\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " Oct 29 23:07:02.609942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.609771 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg\" (UniqueName: \"kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg\") pod \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " Oct 29 23:07:02.609942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.609812 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume\") pod \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\" (UID: \"90656a9d-0221-4d3d-b244-1a0d1ff321a6\") " Oct 29 23:07:02.610079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.609955 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x\" (UniqueName: \"kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x\") pod \"44ab836b-2090-498b-99b5-34f1256aa9b1\" (UID: \"44ab836b-2090-498b-99b5-34f1256aa9b1\") " Oct 29 23:07:02.610174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.610151 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume" pod "44ab836b-2090-498b-99b5-34f1256aa9b1" (UID: "44ab836b-2090-498b-99b5-34f1256aa9b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:02.610299 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.610275 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume" pod "90656a9d-0221-4d3d-b244-1a0d1ff321a6" (UID: "90656a9d-0221-4d3d-b244-1a0d1ff321a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:02.610380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.610363 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ab836b-2090-498b-99b5-34f1256aa9b1-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.612114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.612096 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x" pod "44ab836b-2090-498b-99b5-34f1256aa9b1" (UID: "44ab836b-2090-498b-99b5-34f1256aa9b1"). InnerVolumeSpecName "kube-api-access-2q74x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:02.612180 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.612134 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg" pod "90656a9d-0221-4d3d-b244-1a0d1ff321a6" (UID: "90656a9d-0221-4d3d-b244-1a0d1ff321a6"). InnerVolumeSpecName "kube-api-access-h8thg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:02.711235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.711192 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8thg\" (UniqueName: \"kubernetes.io/projected/90656a9d-0221-4d3d-b244-1a0d1ff321a6-kube-api-access-h8thg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.711235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.711226 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90656a9d-0221-4d3d-b244-1a0d1ff321a6-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.711235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.711242 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q74x\" (UniqueName: \"kubernetes.io/projected/44ab836b-2090-498b-99b5-34f1256aa9b1-kube-api-access-2q74x\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.784401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.784311 5367 generic.go:356] "Generic (PLEG): container finished" podID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerID="58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a" exitCode=0 Oct 29 23:07:02.784401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.784363 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" Oct 29 23:07:02.784631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.784406 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" event={"ID":"44ab836b-2090-498b-99b5-34f1256aa9b1","Type":"ContainerDied","Data":"58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a"} Oct 29 23:07:02.784631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.784462 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8" event={"ID":"44ab836b-2090-498b-99b5-34f1256aa9b1","Type":"ContainerDied","Data":"4c09ec888953f05dff1dbb05afa164de07760b013062096064622792b14060d5"} Oct 29 23:07:02.784631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.784486 5367 scope.go:122] "RemoveContainer" containerID="58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a" Oct 29 23:07:02.787664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.787639 5367 generic.go:356] "Generic (PLEG): container finished" podID="5589ef46-bc4b-4e5f-a94e-369414e4d230" containerID="fcfee064f9c9179972185575e9b08f838860d7c0796d15557262f0bee2606b9f" exitCode=0 Oct 29 23:07:02.787812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.787725 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" event={"ID":"5589ef46-bc4b-4e5f-a94e-369414e4d230","Type":"ContainerDied","Data":"fcfee064f9c9179972185575e9b08f838860d7c0796d15557262f0bee2606b9f"} Oct 29 23:07:02.789108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.789086 5367 generic.go:356] "Generic (PLEG): container finished" podID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerID="bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24" exitCode=0 Oct 29 23:07:02.789228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.789124 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" Oct 29 23:07:02.789228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.789166 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" event={"ID":"90656a9d-0221-4d3d-b244-1a0d1ff321a6","Type":"ContainerDied","Data":"bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24"} Oct 29 23:07:02.789228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.789202 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz" event={"ID":"90656a9d-0221-4d3d-b244-1a0d1ff321a6","Type":"ContainerDied","Data":"7a951b8d4ea88ef936266e04597b620a13df144ba97b6e5d5d6592cc1293fea6"} Oct 29 23:07:02.797735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.797712 5367 scope.go:122] "RemoveContainer" containerID="58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a" Oct 29 23:07:02.798157 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:02.798070 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a\": container with ID starting with 58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a not found: ID does not exist" containerID="58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a" Oct 29 23:07:02.798157 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.798107 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a"} err="failed to get container status \"58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a\": rpc error: code = NotFound desc = could not find container \"58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a\": container with ID starting with 58322629a66149654b87701fde81391f2bbd042608414f09644365cdc61ca84a not found: ID does not exist" Oct 29 23:07:02.798157 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.798131 5367 scope.go:122] "RemoveContainer" containerID="bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24" Oct 29 23:07:02.808990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.808970 5367 scope.go:122] "RemoveContainer" containerID="bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24" Oct 29 23:07:02.809305 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:02.809276 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24\": container with ID starting with bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24 not found: ID does not exist" containerID="bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24" Oct 29 23:07:02.809374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.809318 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24"} err="failed to get container status \"bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24\": rpc error: code = NotFound desc = could not find container \"bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24\": container with ID starting with bc74d762d6bf34249167a660e1a938c535b181499ded77a6fd1244be0c024a24 not found: ID does not exist" Oct 29 23:07:02.843234 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.843205 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz"] Oct 29 23:07:02.846591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.846567 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-92fd86-5d6556c999-7vmpz"] Oct 29 23:07:02.885898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.885866 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8"] Oct 29 23:07:02.890333 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.890301 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-92fd86-6b4699fcfb-tnwj8"] Oct 29 23:07:02.947358 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:02.947330 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:03.013474 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.013444 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca\") pod \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " Oct 29 23:07:03.013624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.013509 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw\" (UniqueName: \"kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw\") pod \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " Oct 29 23:07:03.013694 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.013674 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca\") pod \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\" (UID: \"c181c1f2-c4dc-4687-8f3d-621e0da99b7d\") " Oct 29 23:07:03.014113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.014075 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca" pod "c181c1f2-c4dc-4687-8f3d-621e0da99b7d" (UID: "c181c1f2-c4dc-4687-8f3d-621e0da99b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:03.015541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.015519 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca" pod "c181c1f2-c4dc-4687-8f3d-621e0da99b7d" (UID: "c181c1f2-c4dc-4687-8f3d-621e0da99b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:03.015637 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.015573 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw" pod "c181c1f2-c4dc-4687-8f3d-621e0da99b7d" (UID: "c181c1f2-c4dc-4687-8f3d-621e0da99b7d"). InnerVolumeSpecName "kube-api-access-szwqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:03.115566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.115529 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.115566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.115562 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.115566 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.115572 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwqw\" (UniqueName: \"kubernetes.io/projected/c181c1f2-c4dc-4687-8f3d-621e0da99b7d-kube-api-access-szwqw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.232097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.232063 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" path="/var/lib/kubelet/pods/278233b5-18e7-44bf-940f-76c1e6ee8c42/volumes" Oct 29 23:07:03.232465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.232451 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" path="/var/lib/kubelet/pods/44ab836b-2090-498b-99b5-34f1256aa9b1/volumes" Oct 29 23:07:03.232813 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.232801 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" path="/var/lib/kubelet/pods/90656a9d-0221-4d3d-b244-1a0d1ff321a6/volumes" Oct 29 23:07:03.798988 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.798959 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" Oct 29 23:07:03.799382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.798987 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd" event={"ID":"c181c1f2-c4dc-4687-8f3d-621e0da99b7d","Type":"ContainerDied","Data":"ad3b29c8bc0fc27c4ef389c450fe1827e007313e8c400a3c7babf845443e3905"} Oct 29 23:07:03.799382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.799030 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3b29c8bc0fc27c4ef389c450fe1827e007313e8c400a3c7babf845443e3905" Oct 29 23:07:03.940174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:03.940147 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:04.024691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.024657 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca\") pod \"5589ef46-bc4b-4e5f-a94e-369414e4d230\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " Oct 29 23:07:04.024914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.024757 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr\" (UniqueName: \"kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr\") pod \"5589ef46-bc4b-4e5f-a94e-369414e4d230\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " Oct 29 23:07:04.024914 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.024781 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca\") pod \"5589ef46-bc4b-4e5f-a94e-369414e4d230\" (UID: \"5589ef46-bc4b-4e5f-a94e-369414e4d230\") " Oct 29 23:07:04.025195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.025172 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca" pod "5589ef46-bc4b-4e5f-a94e-369414e4d230" (UID: "5589ef46-bc4b-4e5f-a94e-369414e4d230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:04.026917 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.026896 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr" pod "5589ef46-bc4b-4e5f-a94e-369414e4d230" (UID: "5589ef46-bc4b-4e5f-a94e-369414e4d230"). InnerVolumeSpecName "kube-api-access-c9xkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:04.027019 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.026897 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca" pod "5589ef46-bc4b-4e5f-a94e-369414e4d230" (UID: "5589ef46-bc4b-4e5f-a94e-369414e4d230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:04.126546 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.126512 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:04.126801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.126565 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xkr\" (UniqueName: \"kubernetes.io/projected/5589ef46-bc4b-4e5f-a94e-369414e4d230-kube-api-access-c9xkr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:04.126801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.126575 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5589ef46-bc4b-4e5f-a94e-369414e4d230-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:04.807823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.807788 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" Oct 29 23:07:04.807823 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.807813 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9" event={"ID":"5589ef46-bc4b-4e5f-a94e-369414e4d230","Type":"ContainerDied","Data":"8cebaf11f47dc69b99bc2f1399cf004d85b2610bda833a1d73867e8c24bd1cca"} Oct 29 23:07:04.808278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:04.807862 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cebaf11f47dc69b99bc2f1399cf004d85b2610bda833a1d73867e8c24bd1cca" Oct 29 23:07:05.441911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:05.441869 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" podUID="c8558681-9856-4931-a120-1008a32b6300" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:06.005639 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.005594 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" containerID="cri-o://39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a" gracePeriod=45 Oct 29 23:07:06.261306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.261244 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 23:07:06.348085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348054 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs\") pod \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " Oct 29 23:07:06.348279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348170 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc\" (UniqueName: \"kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc\") pod \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " Oct 29 23:07:06.348279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348207 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home\") pod \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " Oct 29 23:07:06.348279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348247 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir\") pod \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " Oct 29 23:07:06.348279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348279 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm\") pod \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " Oct 29 23:07:06.348517 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache\") pod \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\" (UID: \"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3\") " Oct 29 23:07:06.348968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348520 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home" pod "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" (UID: "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.348968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348566 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir" pod "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" (UID: "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.348968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348733 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache" pod "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" (UID: "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.348968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348810 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.348968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.348834 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.350564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.350535 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm" pod "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" (UID: "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.350669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.350573 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs" pod "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" (UID: "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:06.350669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.350649 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc" pod "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" (UID: "68ad178b-eb24-499b-a9a8-6f1d7e08b8b3"). InnerVolumeSpecName "kube-api-access-n4bmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:06.449724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.449674 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.449724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.449722 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.449940 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.449748 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.449940 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.449760 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bmc\" (UniqueName: \"kubernetes.io/projected/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3-kube-api-access-n4bmc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.824328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.824294 5367 generic.go:356] "Generic (PLEG): container finished" podID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerID="39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a" exitCode=0 Oct 29 23:07:06.824515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.824336 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" Oct 29 23:07:06.824515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.824379 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" event={"ID":"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3","Type":"ContainerDied","Data":"39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a"} Oct 29 23:07:06.824515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.824428 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" event={"ID":"68ad178b-eb24-499b-a9a8-6f1d7e08b8b3","Type":"ContainerDied","Data":"b286037618484ffd8457635a69fb6cead98d4a65fc4d85e9a3c7a061ccbc96e7"} Oct 29 23:07:06.824515 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.824450 5367 scope.go:122] "RemoveContainer" containerID="39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a" Oct 29 23:07:06.835097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.835075 5367 scope.go:122] "RemoveContainer" containerID="39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a" Oct 29 23:07:06.835386 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:06.835363 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a\": container with ID starting with 39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a not found: ID does not exist" containerID="39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a" Oct 29 23:07:06.835478 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.835391 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a"} err="failed to get container status \"39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a\": rpc error: code = NotFound desc = could not find container \"39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a\": container with ID starting with 39a0216f363f8fd4d933bded243699689919a3bcba70bcfef174c2886019752a not found: ID does not exist" Oct 29 23:07:06.839321 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.839298 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb"] Oct 29 23:07:06.875560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.875528 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx"] Oct 29 23:07:06.890242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.890214 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx"] Oct 29 23:07:06.893797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:06.893775 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx"] Oct 29 23:07:07.216396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.216317 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/simulator-w4-4a0287-kserve-787c996f59-7j4mx" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:07.220966 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.220936 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" path="/var/lib/kubelet/pods/68ad178b-eb24-499b-a9a8-6f1d7e08b8b3/volumes" Oct 29 23:07:07.518684 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.518607 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76"] Oct 29 23:07:07.519045 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519031 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5589ef46-bc4b-4e5f-a94e-369414e4d230" containerName="revoke-keys" Oct 29 23:07:07.519093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519047 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5589ef46-bc4b-4e5f-a94e-369414e4d230" containerName="revoke-keys" Oct 29 23:07:07.519093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519062 5367 cpu_manager.go:414] "containerMap: removing container" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" Oct 29 23:07:07.519093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519067 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" Oct 29 23:07:07.519093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519080 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c181c1f2-c4dc-4687-8f3d-621e0da99b7d" containerName="revoke-keys" Oct 29 23:07:07.519093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519086 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c181c1f2-c4dc-4687-8f3d-621e0da99b7d" containerName="revoke-keys" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519100 5367 cpu_manager.go:414] "containerMap: removing container" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerName="payload-pre-processing" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519105 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerName="payload-pre-processing" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519124 5367 cpu_manager.go:414] "containerMap: removing container" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519128 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519136 5367 cpu_manager.go:414] "containerMap: removing container" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerName="payload-processing" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519141 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerName="payload-processing" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519197 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="90656a9d-0221-4d3d-b244-1a0d1ff321a6" containerName="payload-pre-processing" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519209 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="278233b5-18e7-44bf-940f-76c1e6ee8c42" containerName="istio-proxy" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519216 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ab836b-2090-498b-99b5-34f1256aa9b1" containerName="payload-processing" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519222 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5589ef46-bc4b-4e5f-a94e-369414e4d230" containerName="revoke-keys" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519230 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ad178b-eb24-499b-a9a8-6f1d7e08b8b3" containerName="main" Oct 29 23:07:07.519258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.519235 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c181c1f2-c4dc-4687-8f3d-621e0da99b7d" containerName="revoke-keys" Oct 29 23:07:07.522177 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.522159 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.541293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.541254 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6"] Oct 29 23:07:07.542277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.542241 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:07:07.544053 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.543954 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:07:07.544589 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.544517 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:07.545995 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.545972 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.550314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.550290 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:07:07.556484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.556457 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76"] Oct 29 23:07:07.567667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.567636 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6"] Oct 29 23:07:07.662236 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.662197 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.662236 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.662230 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kd8\" (UniqueName: \"kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.662540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.662312 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.662540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.662363 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsj7m\" (UniqueName: \"kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.662540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.662486 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.662540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.662521 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.763562 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.763526 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsj7m\" (UniqueName: \"kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.763789 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.763609 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.763789 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.763680 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.763789 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.763748 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.763789 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.763773 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kd8\" (UniqueName: \"kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.764099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.763834 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.764416 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.764389 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.764544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.764492 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.766176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.766148 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.766176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.766160 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.787083 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.787031 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kd8\" (UniqueName: \"kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8\") pod \"maas-api-revoke-sub-5271d1b8-mlsv6\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:07.787175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.787079 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsj7m\" (UniqueName: \"kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m\") pod \"maas-api-revoke-sub-fe48350f-fxv76\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.836310 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.836282 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:07.871004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:07.870971 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:08.138542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.138513 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76"] Oct 29 23:07:08.139520 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:07:08.139495 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37aeebfd_da1f_4dad_b18c_53babfd8bc38.slice/crio-4bd33c5d39d7795b15f9288a77ba01196a0bfe3b5327c21ebfc120376e9a1897 WatchSource:0}: Error finding container 4bd33c5d39d7795b15f9288a77ba01196a0bfe3b5327c21ebfc120376e9a1897: Status 404 returned error can't find the container with id 4bd33c5d39d7795b15f9288a77ba01196a0bfe3b5327c21ebfc120376e9a1897 Oct 29 23:07:08.189167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.189139 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6"] Oct 29 23:07:08.190258 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:07:08.190232 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c99dd9_39f4_4fc9_ad0f_4644f7702b9f.slice/crio-6e6a20602ade59f2447018d0bf3940e5211cc10d935aeb5da3b6057fdae3f6c7 WatchSource:0}: Error finding container 6e6a20602ade59f2447018d0bf3940e5211cc10d935aeb5da3b6057fdae3f6c7: Status 404 returned error can't find the container with id 6e6a20602ade59f2447018d0bf3940e5211cc10d935aeb5da3b6057fdae3f6c7 Oct 29 23:07:08.842445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.842406 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" event={"ID":"37aeebfd-da1f-4dad-b18c-53babfd8bc38","Type":"ContainerStarted","Data":"9db7080255e1ae18fd9b249547b6d38687af8ad2fd4a5d93404d21037e168bd9"} Oct 29 23:07:08.842949 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.842469 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" event={"ID":"37aeebfd-da1f-4dad-b18c-53babfd8bc38","Type":"ContainerStarted","Data":"4bd33c5d39d7795b15f9288a77ba01196a0bfe3b5327c21ebfc120376e9a1897"} Oct 29 23:07:08.844558 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.844525 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" event={"ID":"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f","Type":"ContainerStarted","Data":"89ed0fa737b8e2231c50776fbaa0883e1aefe80aa0309dc4b6a33e6c153866e3"} Oct 29 23:07:08.844686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.844586 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" event={"ID":"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f","Type":"ContainerStarted","Data":"6e6a20602ade59f2447018d0bf3940e5211cc10d935aeb5da3b6057fdae3f6c7"} Oct 29 23:07:08.888235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.888176 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" podStartSLOduration=1.8881599100000002 podStartE2EDuration="1.88815991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:08.887186481 +0000 UTC m=+3095.781261405" watchObservedRunningTime="2026-10-29 23:07:08.88815991 +0000 UTC m=+3095.782234831" Oct 29 23:07:08.943375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:08.943315 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" podStartSLOduration=1.9432993939999998 podStartE2EDuration="1.943299394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:08.938903338 +0000 UTC m=+3095.832978263" watchObservedRunningTime="2026-10-29 23:07:08.943299394 +0000 UTC m=+3095.837374321" Oct 29 23:07:09.489640 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:09.489605 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:09.489849 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:09.489605 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:09.853295 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:09.853262 5367 generic.go:356] "Generic (PLEG): container finished" podID="37aeebfd-da1f-4dad-b18c-53babfd8bc38" containerID="9db7080255e1ae18fd9b249547b6d38687af8ad2fd4a5d93404d21037e168bd9" exitCode=0 Oct 29 23:07:09.853797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:09.853352 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" event={"ID":"37aeebfd-da1f-4dad-b18c-53babfd8bc38","Type":"ContainerDied","Data":"9db7080255e1ae18fd9b249547b6d38687af8ad2fd4a5d93404d21037e168bd9"} Oct 29 23:07:09.855033 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:09.855009 5367 generic.go:356] "Generic (PLEG): container finished" podID="b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" containerID="89ed0fa737b8e2231c50776fbaa0883e1aefe80aa0309dc4b6a33e6c153866e3" exitCode=0 Oct 29 23:07:09.855156 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:09.855068 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" event={"ID":"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f","Type":"ContainerDied","Data":"89ed0fa737b8e2231c50776fbaa0883e1aefe80aa0309dc4b6a33e6c153866e3"} Oct 29 23:07:11.016362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.016336 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:11.018970 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.018946 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:11.099947 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.099916 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m\" (UniqueName: \"kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m\") pod \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " Oct 29 23:07:11.100153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.099986 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8\" (UniqueName: \"kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8\") pod \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " Oct 29 23:07:11.100153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.100014 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca\") pod \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " Oct 29 23:07:11.100153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.100081 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca\") pod \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " Oct 29 23:07:11.100153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.100099 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca\") pod \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\" (UID: \"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f\") " Oct 29 23:07:11.100153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.100129 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca\") pod \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\" (UID: \"37aeebfd-da1f-4dad-b18c-53babfd8bc38\") " Oct 29 23:07:11.100533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.100501 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca" pod "b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" (UID: "b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:11.100659 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.100504 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca" pod "37aeebfd-da1f-4dad-b18c-53babfd8bc38" (UID: "37aeebfd-da1f-4dad-b18c-53babfd8bc38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:11.102947 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.102919 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m" pod "37aeebfd-da1f-4dad-b18c-53babfd8bc38" (UID: "37aeebfd-da1f-4dad-b18c-53babfd8bc38"). InnerVolumeSpecName "kube-api-access-lsj7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:11.103101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.103077 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca" pod "37aeebfd-da1f-4dad-b18c-53babfd8bc38" (UID: "37aeebfd-da1f-4dad-b18c-53babfd8bc38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:11.103169 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.103119 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8" pod "b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" (UID: "b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f"). InnerVolumeSpecName "kube-api-access-22kd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:11.103348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.103317 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca" pod "b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" (UID: "b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:11.201274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.201185 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsj7m\" (UniqueName: \"kubernetes.io/projected/37aeebfd-da1f-4dad-b18c-53babfd8bc38-kube-api-access-lsj7m\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.201274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.201221 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22kd8\" (UniqueName: \"kubernetes.io/projected/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-kube-api-access-22kd8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.201274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.201233 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.201274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.201243 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.201274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.201253 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.201274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.201262 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aeebfd-da1f-4dad-b18c-53babfd8bc38-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.839791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.839691 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" containerName="payload-processing" containerID="cri-o://a80944571a614d7a5bcdfeb2fdbc2aaab52e4a43fe29586c8fee68580533932d" gracePeriod=25 Oct 29 23:07:11.848871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.848846 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 23:07:11.870538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.870512 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" Oct 29 23:07:11.870725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.870556 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76" event={"ID":"37aeebfd-da1f-4dad-b18c-53babfd8bc38","Type":"ContainerDied","Data":"4bd33c5d39d7795b15f9288a77ba01196a0bfe3b5327c21ebfc120376e9a1897"} Oct 29 23:07:11.870725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.870631 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd33c5d39d7795b15f9288a77ba01196a0bfe3b5327c21ebfc120376e9a1897" Oct 29 23:07:11.872188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.872161 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" Oct 29 23:07:11.872327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.872199 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6" event={"ID":"b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f","Type":"ContainerDied","Data":"6e6a20602ade59f2447018d0bf3940e5211cc10d935aeb5da3b6057fdae3f6c7"} Oct 29 23:07:11.872327 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.872244 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e6a20602ade59f2447018d0bf3940e5211cc10d935aeb5da3b6057fdae3f6c7" Oct 29 23:07:11.874280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.874256 5367 generic.go:356] "Generic (PLEG): container finished" podID="08968dc4-9f0b-4776-b957-b8835c134819" containerID="a41ba8297796640ed4535cfafab76591189f2cd8f7f2aa2e646769906f1a3ccc" exitCode=0 Oct 29 23:07:11.874423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.874392 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" event={"ID":"08968dc4-9f0b-4776-b957-b8835c134819","Type":"ContainerDied","Data":"a41ba8297796640ed4535cfafab76591189f2cd8f7f2aa2e646769906f1a3ccc"} Oct 29 23:07:11.875792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.875771 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerName="payload-pre-processing" containerID="cri-o://8fee0a5e0ce8233c630bdd36268557cf76fd27bf38e138914f311cf673227fea" gracePeriod=25 Oct 29 23:07:11.875912 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.875875 5367 generic.go:356] "Generic (PLEG): container finished" podID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerID="8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09" exitCode=0 Oct 29 23:07:11.875981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.875922 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" Oct 29 23:07:11.876044 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.875973 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" event={"ID":"3136ce3f-d29d-4161-be50-ed68a4fc8cb4","Type":"ContainerDied","Data":"8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09"} Oct 29 23:07:11.876044 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.876022 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz" event={"ID":"3136ce3f-d29d-4161-be50-ed68a4fc8cb4","Type":"ContainerDied","Data":"d904773a86ebbc0dfae004fbbd3b4b8f1dacc0bf9992627e5a13069ea0c7ae7d"} Oct 29 23:07:11.876458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.876055 5367 scope.go:122] "RemoveContainer" containerID="8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09" Oct 29 23:07:11.886602 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.886583 5367 scope.go:122] "RemoveContainer" containerID="8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09" Oct 29 23:07:11.886897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.886877 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 23:07:11.887014 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:11.886942 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09\": container with ID starting with 8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09 not found: ID does not exist" containerID="8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09" Oct 29 23:07:11.887014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.886970 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09"} err="failed to get container status \"8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09\": rpc error: code = NotFound desc = could not find container \"8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09\": container with ID starting with 8ee461ff16014b4466a1d89a5410487ec432b7db35bb18206bae4e2206fffa09 not found: ID does not exist" Oct 29 23:07:11.908422 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.908391 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7\" (UniqueName: \"kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7\") pod \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " Oct 29 23:07:11.908583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.908483 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume\") pod \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\" (UID: \"3136ce3f-d29d-4161-be50-ed68a4fc8cb4\") " Oct 29 23:07:11.908863 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.908840 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume" pod "3136ce3f-d29d-4161-be50-ed68a4fc8cb4" (UID: "3136ce3f-d29d-4161-be50-ed68a4fc8cb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:11.910504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:11.910478 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7" pod "3136ce3f-d29d-4161-be50-ed68a4fc8cb4" (UID: "3136ce3f-d29d-4161-be50-ed68a4fc8cb4"). InnerVolumeSpecName "kube-api-access-cxfq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:12.009882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.009854 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76\" (UniqueName: \"kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76\") pod \"08968dc4-9f0b-4776-b957-b8835c134819\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " Oct 29 23:07:12.010034 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.009913 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume\") pod \"08968dc4-9f0b-4776-b957-b8835c134819\" (UID: \"08968dc4-9f0b-4776-b957-b8835c134819\") " Oct 29 23:07:12.010149 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.010137 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:12.010219 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.010150 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfq7\" (UniqueName: \"kubernetes.io/projected/3136ce3f-d29d-4161-be50-ed68a4fc8cb4-kube-api-access-cxfq7\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:12.010400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.010368 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume" pod "08968dc4-9f0b-4776-b957-b8835c134819" (UID: "08968dc4-9f0b-4776-b957-b8835c134819"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:12.012131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.012105 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76" pod "08968dc4-9f0b-4776-b957-b8835c134819" (UID: "08968dc4-9f0b-4776-b957-b8835c134819"). InnerVolumeSpecName "kube-api-access-b9j76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:12.111794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.111687 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9j76\" (UniqueName: \"kubernetes.io/projected/08968dc4-9f0b-4776-b957-b8835c134819-kube-api-access-b9j76\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:12.111794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.111742 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08968dc4-9f0b-4776-b957-b8835c134819-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:12.224401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.224344 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz"] Oct 29 23:07:12.227929 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.227905 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8f1627-669f7d8ff6-q4hnz"] Oct 29 23:07:12.556024 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.555987 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:12.556211 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.555988 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:12.886155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.886067 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" Oct 29 23:07:12.886155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.886095 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7" event={"ID":"08968dc4-9f0b-4776-b957-b8835c134819","Type":"ContainerDied","Data":"b67dea07144905684801759ef2559a8c73995e7b5cfe1a885ecb2ee7f363b178"} Oct 29 23:07:12.886423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.886162 5367 scope.go:122] "RemoveContainer" containerID="a41ba8297796640ed4535cfafab76591189f2cd8f7f2aa2e646769906f1a3ccc" Oct 29 23:07:12.944109 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.944067 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7"] Oct 29 23:07:12.950479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:12.950450 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8f1627-6d5765975-hv4m7"] Oct 29 23:07:13.163078 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:13.163001 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp"] Oct 29 23:07:13.165630 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:13.165607 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a0287-gkxhp"] Oct 29 23:07:13.220382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:13.220352 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08968dc4-9f0b-4776-b957-b8835c134819" path="/var/lib/kubelet/pods/08968dc4-9f0b-4776-b957-b8835c134819/volumes" Oct 29 23:07:13.220761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:13.220747 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" path="/var/lib/kubelet/pods/3136ce3f-d29d-4161-be50-ed68a4fc8cb4/volumes" Oct 29 23:07:13.221147 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:13.221134 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f13d0b-1743-493f-abed-b311f49f3ea0" path="/var/lib/kubelet/pods/e3f13d0b-1743-493f-abed-b311f49f3ea0/volumes" Oct 29 23:07:15.441967 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:15.441930 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" podUID="c8558681-9856-4931-a120-1008a32b6300" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:16.203684 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.203647 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" containerID="cri-o://40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb" gracePeriod=45 Oct 29 23:07:16.234481 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.234439 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" containerID="cri-o://27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1" gracePeriod=45 Oct 29 23:07:16.502725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.502680 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 23:07:16.532176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.532088 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 23:07:16.659478 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659444 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs\") pod \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " Oct 29 23:07:16.659664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659530 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm\") pod \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " Oct 29 23:07:16.659664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659579 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home\") pod \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " Oct 29 23:07:16.659664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659620 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6\" (UniqueName: \"kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6\") pod \"c7a9c642-24bd-4b19-a233-0f635554b1dc\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " Oct 29 23:07:16.659664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659661 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs\") pod \"c7a9c642-24bd-4b19-a233-0f635554b1dc\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " Oct 29 23:07:16.659932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659694 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn\" (UniqueName: \"kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn\") pod \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " Oct 29 23:07:16.659932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659782 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm\") pod \"c7a9c642-24bd-4b19-a233-0f635554b1dc\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " Oct 29 23:07:16.659932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659811 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir\") pod \"c7a9c642-24bd-4b19-a233-0f635554b1dc\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " Oct 29 23:07:16.659932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659839 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home" pod "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" (UID: "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.659932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659856 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home\") pod \"c7a9c642-24bd-4b19-a233-0f635554b1dc\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " Oct 29 23:07:16.659932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659921 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache\") pod \"c7a9c642-24bd-4b19-a233-0f635554b1dc\" (UID: \"c7a9c642-24bd-4b19-a233-0f635554b1dc\") " Oct 29 23:07:16.660237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.659963 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache\") pod \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " Oct 29 23:07:16.660237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660000 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir\") pod \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\" (UID: \"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8\") " Oct 29 23:07:16.660237 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660094 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir" pod "c7a9c642-24bd-4b19-a233-0f635554b1dc" (UID: "c7a9c642-24bd-4b19-a233-0f635554b1dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.660403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660324 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache" pod "c7a9c642-24bd-4b19-a233-0f635554b1dc" (UID: "c7a9c642-24bd-4b19-a233-0f635554b1dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.660468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660386 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home" pod "c7a9c642-24bd-4b19-a233-0f635554b1dc" (UID: "c7a9c642-24bd-4b19-a233-0f635554b1dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.660528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660470 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.660528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660465 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache" pod "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" (UID: "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.660528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660513 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.660669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660529 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.660669 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.660538 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir" pod "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" (UID: "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.662067 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.662039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6" pod "c7a9c642-24bd-4b19-a233-0f635554b1dc" (UID: "c7a9c642-24bd-4b19-a233-0f635554b1dc"). InnerVolumeSpecName "kube-api-access-8szn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:16.662208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.662196 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs" pod "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" (UID: "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:16.662279 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.662250 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm" pod "c7a9c642-24bd-4b19-a233-0f635554b1dc" (UID: "c7a9c642-24bd-4b19-a233-0f635554b1dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.662335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.662325 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm" pod "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" (UID: "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.662469 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.662449 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn" pod "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" (UID: "72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8"). InnerVolumeSpecName "kube-api-access-9dvgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:16.662605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.662573 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs" pod "c7a9c642-24bd-4b19-a233-0f635554b1dc" (UID: "c7a9c642-24bd-4b19-a233-0f635554b1dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761608 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761639 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761651 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761660 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761667 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761675 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szn6\" (UniqueName: \"kubernetes.io/projected/c7a9c642-24bd-4b19-a233-0f635554b1dc-kube-api-access-8szn6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761686 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a9c642-24bd-4b19-a233-0f635554b1dc-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761694 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvgn\" (UniqueName: \"kubernetes.io/projected/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8-kube-api-access-9dvgn\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.761707 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.761713 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a9c642-24bd-4b19-a233-0f635554b1dc-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.920287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.920250 5367 generic.go:356] "Generic (PLEG): container finished" podID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerID="27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1" exitCode=0 Oct 29 23:07:16.920477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.920293 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" Oct 29 23:07:16.920477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.920324 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" event={"ID":"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8","Type":"ContainerDied","Data":"27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1"} Oct 29 23:07:16.920477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.920372 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9" event={"ID":"72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8","Type":"ContainerDied","Data":"48292ec89a75d44edea50bfea8fc35c49c0caebed9de9a99ef401fede4cae5db"} Oct 29 23:07:16.920477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.920390 5367 scope.go:122] "RemoveContainer" containerID="27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1" Oct 29 23:07:16.922116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.922092 5367 generic.go:356] "Generic (PLEG): container finished" podID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerID="40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb" exitCode=0 Oct 29 23:07:16.922254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.922160 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" Oct 29 23:07:16.922254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.922180 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" event={"ID":"c7a9c642-24bd-4b19-a233-0f635554b1dc","Type":"ContainerDied","Data":"40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb"} Oct 29 23:07:16.922254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.922219 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq" event={"ID":"c7a9c642-24bd-4b19-a233-0f635554b1dc","Type":"ContainerDied","Data":"9d2e2f3251228f903bf9343784234e8d286f4cbe476728beec1b6e77c9ef2ef4"} Oct 29 23:07:16.931885 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.931752 5367 scope.go:122] "RemoveContainer" containerID="27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1" Oct 29 23:07:16.932112 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:16.932087 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1\": container with ID starting with 27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1 not found: ID does not exist" containerID="27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1" Oct 29 23:07:16.932179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.932125 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1"} err="failed to get container status \"27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1\": rpc error: code = NotFound desc = could not find container \"27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1\": container with ID starting with 27986de6b387bbc8a89aec74fb68271c9bee3a905928c70a87be84d941ff53d1 not found: ID does not exist" Oct 29 23:07:16.932179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.932146 5367 scope.go:122] "RemoveContainer" containerID="40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb" Oct 29 23:07:16.941747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.941729 5367 scope.go:122] "RemoveContainer" containerID="40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb" Oct 29 23:07:16.942011 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:16.941992 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb\": container with ID starting with 40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb not found: ID does not exist" containerID="40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb" Oct 29 23:07:16.942077 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.942018 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb"} err="failed to get container status \"40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb\": rpc error: code = NotFound desc = could not find container \"40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb\": container with ID starting with 40942310050ba01a02f2c0c3b7d5daa27211d2f87c9686bd0839b165f5399bdb not found: ID does not exist" Oct 29 23:07:16.977265 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.977237 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9"] Oct 29 23:07:16.991095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:16.991063 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/simulator-w3-95672c-kserve-5957b56dbd-xpxs9"] Oct 29 23:07:17.027980 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:17.027888 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq"] Oct 29 23:07:17.049103 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:17.049075 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-w3-95672c-kserve-cc746d98d-k49xq"] Oct 29 23:07:17.221553 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:17.221513 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" path="/var/lib/kubelet/pods/72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8/volumes" Oct 29 23:07:17.222030 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:17.222009 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" path="/var/lib/kubelet/pods/c7a9c642-24bd-4b19-a233-0f635554b1dc/volumes" Oct 29 23:07:20.049479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:20.049436 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk"] Oct 29 23:07:20.050014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:20.049808 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" containerID="cri-o://ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0" gracePeriod=30 Oct 29 23:07:20.734619 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:20.734581 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp"] Oct 29 23:07:20.737693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:20.737664 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-95672c-cbkfp"] Oct 29 23:07:21.221175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:21.221139 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf06c62-1105-4ca9-91ce-493e15b0cfa5" path="/var/lib/kubelet/pods/daf06c62-1105-4ca9-91ce-493e15b0cfa5/volumes" Oct 29 23:07:22.260832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.260803 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 23:07:22.335632 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:22.335591 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:07:22.420129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.420088 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt\" (UniqueName: \"kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt\") pod \"c8558681-9856-4931-a120-1008a32b6300\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " Oct 29 23:07:22.420129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.420133 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume\") pod \"c8558681-9856-4931-a120-1008a32b6300\" (UID: \"c8558681-9856-4931-a120-1008a32b6300\") " Oct 29 23:07:22.420554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.420527 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume" pod "c8558681-9856-4931-a120-1008a32b6300" (UID: "c8558681-9856-4931-a120-1008a32b6300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:22.422441 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.422406 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt" pod "c8558681-9856-4931-a120-1008a32b6300" (UID: "c8558681-9856-4931-a120-1008a32b6300"). InnerVolumeSpecName "kube-api-access-klfqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:22.521415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.521330 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfqt\" (UniqueName: \"kubernetes.io/projected/c8558681-9856-4931-a120-1008a32b6300-kube-api-access-klfqt\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.521415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.521360 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8558681-9856-4931-a120-1008a32b6300-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.556185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.556144 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:22.556346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.556160 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:22.971058 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.971024 5367 generic.go:356] "Generic (PLEG): container finished" podID="c8558681-9856-4931-a120-1008a32b6300" containerID="b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e" exitCode=0 Oct 29 23:07:22.971250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.971071 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" Oct 29 23:07:22.971250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.971082 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" event={"ID":"c8558681-9856-4931-a120-1008a32b6300","Type":"ContainerDied","Data":"b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e"} Oct 29 23:07:22.971250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.971130 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9" event={"ID":"c8558681-9856-4931-a120-1008a32b6300","Type":"ContainerDied","Data":"0cb1affb54dba8da45eea62b304d05b18bc9c2b85839ed5d76e688a220937db1"} Oct 29 23:07:22.971250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.971154 5367 scope.go:122] "RemoveContainer" containerID="b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e" Oct 29 23:07:22.981547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.981529 5367 scope.go:122] "RemoveContainer" containerID="b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e" Oct 29 23:07:22.981861 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:22.981840 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e\": container with ID starting with b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e not found: ID does not exist" containerID="b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e" Oct 29 23:07:22.981929 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:22.981872 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e"} err="failed to get container status \"b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e\": rpc error: code = NotFound desc = could not find container \"b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e\": container with ID starting with b0e2db670b273d94ed0c9a7f0254d30df9d27e3cf58d55513d0195bcedbf7e7e not found: ID does not exist" Oct 29 23:07:23.031401 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:23.031357 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9"] Oct 29 23:07:23.044184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:23.044153 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a0287-cc6d67c5d-m5kn9"] Oct 29 23:07:23.220442 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:23.220406 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8558681-9856-4931-a120-1008a32b6300" path="/var/lib/kubelet/pods/c8558681-9856-4931-a120-1008a32b6300/volumes" Oct 29 23:07:25.317929 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.317903 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 23:07:25.348325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348253 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348419 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348568 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348508 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348577 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348606 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348654 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348667 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.348876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348757 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo\") pod \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\" (UID: \"3b7fed1d-bb2f-44e3-b1af-6043e5208157\") " Oct 29 23:07:25.348876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348762 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:25.349054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.348943 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.349158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.349136 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.349224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.349156 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.349224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.349178 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.349224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.349193 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.349224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.349200 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.350628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.350595 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:25.350762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.350645 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.350762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.350729 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "kube-api-access-lc5dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:25.350961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.350945 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo" pod "3b7fed1d-bb2f-44e3-b1af-6043e5208157" (UID: "3b7fed1d-bb2f-44e3-b1af-6043e5208157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:25.450118 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.450080 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.450118 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.450111 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.450118 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.450124 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.450360 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.450136 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5dj\" (UniqueName: \"kubernetes.io/projected/3b7fed1d-bb2f-44e3-b1af-6043e5208157-kube-api-access-lc5dj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.450360 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.450148 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.450360 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.450160 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7fed1d-bb2f-44e3-b1af-6043e5208157-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.993477 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.993443 5367 generic.go:356] "Generic (PLEG): container finished" podID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerID="ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0" exitCode=0 Oct 29 23:07:25.993670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.993494 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" Oct 29 23:07:25.993670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.993534 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" event={"ID":"3b7fed1d-bb2f-44e3-b1af-6043e5208157","Type":"ContainerDied","Data":"ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0"} Oct 29 23:07:25.993670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.993589 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk" event={"ID":"3b7fed1d-bb2f-44e3-b1af-6043e5208157","Type":"ContainerDied","Data":"60fc0f745f1f920a37c453f16006996e4129708a4c813c01f0b286ac86b96b79"} Oct 29 23:07:25.993670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:25.993617 5367 scope.go:122] "RemoveContainer" containerID="ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0" Oct 29 23:07:26.004011 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.003991 5367 scope.go:122] "RemoveContainer" containerID="e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11" Oct 29 23:07:26.012955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.012936 5367 scope.go:122] "RemoveContainer" containerID="ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0" Oct 29 23:07:26.013206 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:26.013187 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0\": container with ID starting with ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0 not found: ID does not exist" containerID="ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0" Oct 29 23:07:26.013244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.013216 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0"} err="failed to get container status \"ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0\": rpc error: code = NotFound desc = could not find container \"ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0\": container with ID starting with ee1d5a6047e65df7e1d5ce6af8f94e06b335084d92d3f7f0d6ae70cfb1823ca0 not found: ID does not exist" Oct 29 23:07:26.013244 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.013232 5367 scope.go:122] "RemoveContainer" containerID="e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11" Oct 29 23:07:26.013439 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:26.013424 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11\": container with ID starting with e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11 not found: ID does not exist" containerID="e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11" Oct 29 23:07:26.013484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.013444 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11"} err="failed to get container status \"e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11\": rpc error: code = NotFound desc = could not find container \"e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11\": container with ID starting with e86beec3dbb140b9069b4d2e109da7e870baaef61d2e40b7bc73bbc2bea7ba11 not found: ID does not exist" Oct 29 23:07:26.064921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.064884 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk"] Oct 29 23:07:26.082289 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:26.082248 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-qhsrk"] Oct 29 23:07:27.221183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:27.221148 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" path="/var/lib/kubelet/pods/3b7fed1d-bb2f-44e3-b1af-6043e5208157/volumes" Oct 29 23:07:28.167822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:28.167788 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc"] Oct 29 23:07:28.168302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:28.168267 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" containerID="cri-o://9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072" gracePeriod=30 Oct 29 23:07:32.038586 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.038552 5367 generic.go:356] "Generic (PLEG): container finished" podID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerID="8fee0a5e0ce8233c630bdd36268557cf76fd27bf38e138914f311cf673227fea" exitCode=0 Oct 29 23:07:32.039007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.038764 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" event={"ID":"c0b776b7-051e-4b07-8c62-c76f2c210ad9","Type":"ContainerDied","Data":"8fee0a5e0ce8233c630bdd36268557cf76fd27bf38e138914f311cf673227fea"} Oct 29 23:07:32.041132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.041108 5367 generic.go:356] "Generic (PLEG): container finished" podID="acffe8f2-1085-40f6-9093-f916144f64c1" containerID="a80944571a614d7a5bcdfeb2fdbc2aaab52e4a43fe29586c8fee68580533932d" exitCode=0 Oct 29 23:07:32.041262 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.041237 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" event={"ID":"acffe8f2-1085-40f6-9093-f916144f64c1","Type":"ContainerDied","Data":"a80944571a614d7a5bcdfeb2fdbc2aaab52e4a43fe29586c8fee68580533932d"} Oct 29 23:07:32.151396 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.151367 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 23:07:32.154152 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.154128 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 23:07:32.212094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.212062 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume\") pod \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " Oct 29 23:07:32.212278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.212109 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb\" (UniqueName: \"kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb\") pod \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\" (UID: \"c0b776b7-051e-4b07-8c62-c76f2c210ad9\") " Oct 29 23:07:32.212278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.212162 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w\" (UniqueName: \"kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w\") pod \"acffe8f2-1085-40f6-9093-f916144f64c1\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " Oct 29 23:07:32.212278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.212244 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume\") pod \"acffe8f2-1085-40f6-9093-f916144f64c1\" (UID: \"acffe8f2-1085-40f6-9093-f916144f64c1\") " Oct 29 23:07:32.212654 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.212487 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume" pod "c0b776b7-051e-4b07-8c62-c76f2c210ad9" (UID: "c0b776b7-051e-4b07-8c62-c76f2c210ad9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:32.212773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.212676 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume" pod "acffe8f2-1085-40f6-9093-f916144f64c1" (UID: "acffe8f2-1085-40f6-9093-f916144f64c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:32.214300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.214271 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb" pod "c0b776b7-051e-4b07-8c62-c76f2c210ad9" (UID: "c0b776b7-051e-4b07-8c62-c76f2c210ad9"). InnerVolumeSpecName "kube-api-access-4rnpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:32.214449 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.214423 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w" pod "acffe8f2-1085-40f6-9093-f916144f64c1" (UID: "acffe8f2-1085-40f6-9093-f916144f64c1"). InnerVolumeSpecName "kube-api-access-dx85w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:32.313469 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.313431 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acffe8f2-1085-40f6-9093-f916144f64c1-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:32.313469 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.313464 5367 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b776b7-051e-4b07-8c62-c76f2c210ad9-plugins-config-volume\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:32.313469 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.313477 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rnpb\" (UniqueName: \"kubernetes.io/projected/c0b776b7-051e-4b07-8c62-c76f2c210ad9-kube-api-access-4rnpb\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:32.313777 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:32.313489 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx85w\" (UniqueName: \"kubernetes.io/projected/acffe8f2-1085-40f6-9093-f916144f64c1-kube-api-access-dx85w\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.049865 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.049836 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" Oct 29 23:07:33.049865 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.049851 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb" event={"ID":"acffe8f2-1085-40f6-9093-f916144f64c1","Type":"ContainerDied","Data":"05abbf855d24e7ee73263faa8a32637ddd8570a4fe524104cd39ad305a6484b9"} Oct 29 23:07:33.050421 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.049933 5367 scope.go:122] "RemoveContainer" containerID="a80944571a614d7a5bcdfeb2fdbc2aaab52e4a43fe29586c8fee68580533932d" Oct 29 23:07:33.051527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.051507 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" Oct 29 23:07:33.051634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.051535 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx" event={"ID":"c0b776b7-051e-4b07-8c62-c76f2c210ad9","Type":"ContainerDied","Data":"14c1557ec1541a0b2c840ebecf867f85399831b92b38cc870280d1f0653b5fb4"} Oct 29 23:07:33.062214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.062193 5367 scope.go:122] "RemoveContainer" containerID="8fee0a5e0ce8233c630bdd36268557cf76fd27bf38e138914f311cf673227fea" Oct 29 23:07:33.116615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.116579 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx"] Oct 29 23:07:33.130376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.130341 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-95672c-5bcdd68f75-ss8xx"] Oct 29 23:07:33.168190 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.168162 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb"] Oct 29 23:07:33.175693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.175663 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-95672c-7dcb6ff684-rrcdb"] Oct 29 23:07:33.220278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.220243 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" path="/var/lib/kubelet/pods/acffe8f2-1085-40f6-9093-f916144f64c1/volumes" Oct 29 23:07:33.220769 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.220747 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" path="/var/lib/kubelet/pods/c0b776b7-051e-4b07-8c62-c76f2c210ad9/volumes" Oct 29 23:07:33.413864 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.413841 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 23:07:33.526521 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526484 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526537 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526590 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526645 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526732 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526787 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526786 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.526881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526830 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.526881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526863 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.527173 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526928 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket\") pod \"819c85fe-d774-4609-8084-199ee2718352\" (UID: \"819c85fe-d774-4609-8084-199ee2718352\") " Oct 29 23:07:33.527173 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.526991 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.527291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.527246 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.527291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.527263 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.527414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.527311 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:33.527414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.527321 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.527527 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.527468 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.529062 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.529034 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.529212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.529151 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:33.529288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.529273 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:33.529352 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.529295 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5" pod "819c85fe-d774-4609-8084-199ee2718352" (UID: "819c85fe-d774-4609-8084-199ee2718352"). InnerVolumeSpecName "kube-api-access-gzcm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628178 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819c85fe-d774-4609-8084-199ee2718352-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628212 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628220 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628228 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628235 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819c85fe-d774-4609-8084-199ee2718352-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628255 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819c85fe-d774-4609-8084-199ee2718352-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.628263 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:33.628264 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzcm5\" (UniqueName: \"kubernetes.io/projected/819c85fe-d774-4609-8084-199ee2718352-kube-api-access-gzcm5\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:34.060044 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.060011 5367 generic.go:356] "Generic (PLEG): container finished" podID="819c85fe-d774-4609-8084-199ee2718352" containerID="9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072" exitCode=0 Oct 29 23:07:34.060044 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.060054 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" Oct 29 23:07:34.060597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.060094 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" event={"ID":"819c85fe-d774-4609-8084-199ee2718352","Type":"ContainerDied","Data":"9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072"} Oct 29 23:07:34.060597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.060148 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc" event={"ID":"819c85fe-d774-4609-8084-199ee2718352","Type":"ContainerDied","Data":"24fa8933b3433cdf74f49b9ea99675e22d2dd5b9ac03151972b647e617dbcc3a"} Oct 29 23:07:34.060597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.060181 5367 scope.go:122] "RemoveContainer" containerID="9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072" Oct 29 23:07:34.071030 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.071014 5367 scope.go:122] "RemoveContainer" containerID="9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072" Oct 29 23:07:34.071322 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:34.071304 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072\": container with ID starting with 9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072 not found: ID does not exist" containerID="9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072" Oct 29 23:07:34.071379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.071334 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072"} err="failed to get container status \"9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072\": rpc error: code = NotFound desc = could not find container \"9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072\": container with ID starting with 9792a5bba03d939f2555e2ac64642cdcd1f2e7713e3e886f31bd3f4cb7a61072 not found: ID does not exist" Oct 29 23:07:34.117891 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.117857 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc"] Oct 29 23:07:34.131987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:34.131951 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-ggbqc"] Oct 29 23:07:35.220969 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:35.220938 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819c85fe-d774-4609-8084-199ee2718352" path="/var/lib/kubelet/pods/819c85fe-d774-4609-8084-199ee2718352/volumes" Oct 29 23:07:38.028890 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:38.028856 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h"] Oct 29 23:07:38.042866 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:38.042837 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f60c83-7854h"] Oct 29 23:07:38.044882 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:38.044869 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78"] Oct 29 23:07:38.046621 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:38.046599 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842fd636-rgr78"] Oct 29 23:07:39.220094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:39.220051 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1421c097-e373-4efa-9b49-256eb3438c6a" path="/var/lib/kubelet/pods/1421c097-e373-4efa-9b49-256eb3438c6a/volumes" Oct 29 23:07:39.220659 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:39.220637 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294ad7f3-5730-49a7-85d0-19ff1b8b13b8" path="/var/lib/kubelet/pods/294ad7f3-5730-49a7-85d0-19ff1b8b13b8/volumes" Oct 29 23:07:39.644690 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:39.644656 5367 scope.go:122] "RemoveContainer" containerID="c1d777dc103191b16ede5864b8049f85ef093941f7c30edf4e709e6c0f5cacc8" Oct 29 23:07:39.654255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:39.654235 5367 scope.go:122] "RemoveContainer" containerID="5cc46314cb4fffd0ea606657276b007fa57f54987ad89548d56f66a96dab904f" Oct 29 23:07:39.663192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:39.663169 5367 scope.go:122] "RemoveContainer" containerID="de7908022c8478194a2a81e26a9ed4cc0d5383d658ada8f8621567d201702590" Oct 29 23:07:49.330229 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330192 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw"] Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330574 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerName="payload-pre-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330586 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerName="payload-pre-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330599 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c8558681-9856-4931-a120-1008a32b6300" containerName="payload-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330604 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8558681-9856-4931-a120-1008a32b6300" containerName="payload-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330629 5367 cpu_manager.go:414] "containerMap: removing container" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330637 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330643 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerName="payload-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330648 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerName="payload-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330657 5367 cpu_manager.go:414] "containerMap: removing container" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" containerName="payload-processing" Oct 29 23:07:49.330657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330662 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" containerName="payload-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330669 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" containerName="revoke-keys" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330673 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" containerName="revoke-keys" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330680 5367 cpu_manager.go:414] "containerMap: removing container" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330685 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330693 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330716 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330729 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330733 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330741 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330745 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330753 5367 cpu_manager.go:414] "containerMap: removing container" podUID="37aeebfd-da1f-4dad-b18c-53babfd8bc38" containerName="revoke-keys" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330757 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37aeebfd-da1f-4dad-b18c-53babfd8bc38" containerName="revoke-keys" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330763 5367 cpu_manager.go:414] "containerMap: removing container" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330767 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330830 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b776b7-051e-4b07-8c62-c76f2c210ad9" containerName="payload-pre-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330837 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8558681-9856-4931-a120-1008a32b6300" containerName="payload-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330847 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a3568c-59dd-4b3c-9e3f-3e9ca926c3b8" containerName="main" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330852 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3136ce3f-d29d-4161-be50-ed68a4fc8cb4" containerName="payload-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330860 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="acffe8f2-1085-40f6-9093-f916144f64c1" containerName="payload-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330870 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" containerName="revoke-keys" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330878 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="08968dc4-9f0b-4776-b957-b8835c134819" containerName="payload-pre-processing" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330883 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330888 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7fed1d-bb2f-44e3-b1af-6043e5208157" containerName="istio-proxy" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330896 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a9c642-24bd-4b19-a233-0f635554b1dc" containerName="main" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330901 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="37aeebfd-da1f-4dad-b18c-53babfd8bc38" containerName="revoke-keys" Oct 29 23:07:49.331063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.330906 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="819c85fe-d774-4609-8084-199ee2718352" containerName="istio-proxy" Oct 29 23:07:49.339072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.339050 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.373897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.373869 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw"] Oct 29 23:07:49.473569 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473532 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.473826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473597 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4s2d\" (UniqueName: \"kubernetes.io/projected/d70fe24b-0114-4d0a-9739-e057bac7f836-kube-api-access-d4s2d\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.473826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473686 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70fe24b-0114-4d0a-9739-e057bac7f836-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.473826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473805 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.474014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473869 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.474014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473886 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.474014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473904 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.474014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.473956 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.474166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.474079 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.574675 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.574637 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.574911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.574826 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.574911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.574866 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.574911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.574889 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.574911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.574909 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.574940 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575000 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575079 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4s2d\" (UniqueName: \"kubernetes.io/projected/d70fe24b-0114-4d0a-9739-e057bac7f836-kube-api-access-d4s2d\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575104 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70fe24b-0114-4d0a-9739-e057bac7f836-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575152 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575257 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575394 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575414 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.575935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.575916 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70fe24b-0114-4d0a-9739-e057bac7f836-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.577277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.577255 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.577393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.577376 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.596256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.596188 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70fe24b-0114-4d0a-9739-e057bac7f836-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.596414 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.596265 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4s2d\" (UniqueName: \"kubernetes.io/projected/d70fe24b-0114-4d0a-9739-e057bac7f836-kube-api-access-d4s2d\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kthhw\" (UID: \"d70fe24b-0114-4d0a-9739-e057bac7f836\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.653398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.653356 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:49.969209 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.969184 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw"] Oct 29 23:07:49.972498 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:07:49.972465 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70fe24b_0114_4d0a_9739_e057bac7f836.slice/crio-2b9438095897705c56eb39b3c37b78806df7300ee483c83209fc5b0c7bd8d501 WatchSource:0}: Error finding container 2b9438095897705c56eb39b3c37b78806df7300ee483c83209fc5b0c7bd8d501: Status 404 returned error can't find the container with id 2b9438095897705c56eb39b3c37b78806df7300ee483c83209fc5b0c7bd8d501 Oct 29 23:07:49.975446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.975409 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:07:49.975533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.975478 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:07:49.975533 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:49.975505 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:07:50.175899 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:50.175853 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" event={"ID":"d70fe24b-0114-4d0a-9739-e057bac7f836","Type":"ContainerStarted","Data":"f4109c74f7408b4c501b5c200cd3706c8a737f1b71b2de20b1f1fe393007470c"} Oct 29 23:07:50.176114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:50.175915 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" event={"ID":"d70fe24b-0114-4d0a-9739-e057bac7f836","Type":"ContainerStarted","Data":"2b9438095897705c56eb39b3c37b78806df7300ee483c83209fc5b0c7bd8d501"} Oct 29 23:07:50.227014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:50.226907 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podStartSLOduration=1.226892376 podStartE2EDuration="1.226892376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:50.225544488 +0000 UTC m=+3137.119619414" watchObservedRunningTime="2026-10-29 23:07:50.226892376 +0000 UTC m=+3137.120967297" Oct 29 23:07:50.653775 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:50.653694 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:07:51.677151 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:51.677110 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body= Oct 29 23:07:51.677802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:51.677764 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" Oct 29 23:07:52.330116 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:07:52.330080 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:07:52.663264 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:52.663165 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body= Oct 29 23:07:52.663522 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:52.663491 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" Oct 29 23:07:53.654002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.653962 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body= Oct 29 23:07:53.654678 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.654641 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" Oct 29 23:07:53.933138 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.933043 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q"] Oct 29 23:07:53.936355 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.936329 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:53.984904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.969413 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q"] Oct 29 23:07:53.984904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.978566 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:07:53.984904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.978738 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:07:53.984904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.978785 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:53.984904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:53.978841 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:07:54.021339 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.021294 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.021543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.021350 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xbc\" (UniqueName: \"kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.021543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.021445 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.122459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.122405 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.122633 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.122471 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xbc\" (UniqueName: \"kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.122633 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.122521 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.123165 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.123143 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.124942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.124916 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.144679 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.144638 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xbc\" (UniqueName: \"kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc\") pod \"maas-api-revoke-sub-85237e45-6cn9q\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.249755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.249641 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:54.580317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.580283 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q"] Oct 29 23:07:54.583687 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:07:54.583647 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1827a175_3964_4836_8607_420e43f04878.slice/crio-0ee1930a61cc012e28513ec2af2d93a178773ac0ba0c305d9d5963833a6ed71d WatchSource:0}: Error finding container 0ee1930a61cc012e28513ec2af2d93a178773ac0ba0c305d9d5963833a6ed71d: Status 404 returned error can't find the container with id 0ee1930a61cc012e28513ec2af2d93a178773ac0ba0c305d9d5963833a6ed71d Oct 29 23:07:54.674723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.674169 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body= Oct 29 23:07:54.674723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:54.674240 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" Oct 29 23:07:55.219071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:55.219039 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" event={"ID":"1827a175-3964-4836-8607-420e43f04878","Type":"ContainerStarted","Data":"1a54c2b7c7473836d50078bb2bf7bf4b17df94d9593512ba52d676ac141bc641"} Oct 29 23:07:55.219266 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:55.219090 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" event={"ID":"1827a175-3964-4836-8607-420e43f04878","Type":"ContainerStarted","Data":"0ee1930a61cc012e28513ec2af2d93a178773ac0ba0c305d9d5963833a6ed71d"} Oct 29 23:07:55.279785 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:55.275555 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" podStartSLOduration=2.275535271 podStartE2EDuration="2.275535271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:55.274122994 +0000 UTC m=+3142.168197918" watchObservedRunningTime="2026-10-29 23:07:55.275535271 +0000 UTC m=+3142.169610193" Oct 29 23:07:55.673724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:55.659101 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body= Oct 29 23:07:55.673724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:55.659165 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" Oct 29 23:07:56.222876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:56.222841 5367 generic.go:356] "Generic (PLEG): container finished" podID="1827a175-3964-4836-8607-420e43f04878" containerID="1a54c2b7c7473836d50078bb2bf7bf4b17df94d9593512ba52d676ac141bc641" exitCode=0 Oct 29 23:07:56.223347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:56.222924 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" event={"ID":"1827a175-3964-4836-8607-420e43f04878","Type":"ContainerDied","Data":"1a54c2b7c7473836d50078bb2bf7bf4b17df94d9593512ba52d676ac141bc641"} Oct 29 23:07:56.654622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:56.654580 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body= Oct 29 23:07:56.654836 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:56.654657 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" Oct 29 23:07:57.376185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.376153 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:57.456580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.456545 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca\") pod \"1827a175-3964-4836-8607-420e43f04878\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " Oct 29 23:07:57.456794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.456598 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca\") pod \"1827a175-3964-4836-8607-420e43f04878\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " Oct 29 23:07:57.456794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.456667 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc\" (UniqueName: \"kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc\") pod \"1827a175-3964-4836-8607-420e43f04878\" (UID: \"1827a175-3964-4836-8607-420e43f04878\") " Oct 29 23:07:57.457046 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.457023 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca" pod "1827a175-3964-4836-8607-420e43f04878" (UID: "1827a175-3964-4836-8607-420e43f04878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:57.457129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.457113 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1827a175-3964-4836-8607-420e43f04878-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.458913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.458881 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc" pod "1827a175-3964-4836-8607-420e43f04878" (UID: "1827a175-3964-4836-8607-420e43f04878"). InnerVolumeSpecName "kube-api-access-89xbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:57.458913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.458882 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca" pod "1827a175-3964-4836-8607-420e43f04878" (UID: "1827a175-3964-4836-8607-420e43f04878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:57.558079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.558031 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89xbc\" (UniqueName: \"kubernetes.io/projected/1827a175-3964-4836-8607-420e43f04878-kube-api-access-89xbc\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.558079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:57.558076 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1827a175-3964-4836-8607-420e43f04878-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:07:58.239076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:58.239047 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" Oct 29 23:07:58.239306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:58.239091 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q" event={"ID":"1827a175-3964-4836-8607-420e43f04878","Type":"ContainerDied","Data":"0ee1930a61cc012e28513ec2af2d93a178773ac0ba0c305d9d5963833a6ed71d"} Oct 29 23:07:58.239306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:58.239153 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee1930a61cc012e28513ec2af2d93a178773ac0ba0c305d9d5963833a6ed71d" Oct 29 23:07:58.654573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:58.654528 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:07:58.655147 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:58.654615 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:59.695918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:59.695870 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:07:59.696372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:07:59.695954 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:00.307338 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:00.307306 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:08:00.307854 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:00.307814 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:08:00.308617 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:00.308596 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" Oct 29 23:08:04.381298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.381257 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx"] Oct 29 23:08:04.381754 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.381653 5367 cpu_manager.go:414] "containerMap: removing container" podUID="1827a175-3964-4836-8607-420e43f04878" containerName="revoke-keys" Oct 29 23:08:04.381754 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.381664 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1827a175-3964-4836-8607-420e43f04878" containerName="revoke-keys" Oct 29 23:08:04.381754 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.381744 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="1827a175-3964-4836-8607-420e43f04878" containerName="revoke-keys" Oct 29 23:08:04.384795 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.384773 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.417735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.417686 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx"] Oct 29 23:08:04.527429 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527390 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527440 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v9td\" (UniqueName: \"kubernetes.io/projected/2bcf97eb-0392-4819-8114-e0c06d076ca6-kube-api-access-2v9td\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527461 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcf97eb-0392-4819-8114-e0c06d076ca6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527589 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527653 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527682 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527727 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527821 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.527954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.527855 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.628893 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.628860 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.628928 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.628970 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v9td\" (UniqueName: \"kubernetes.io/projected/2bcf97eb-0392-4819-8114-e0c06d076ca6-kube-api-access-2v9td\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.628993 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcf97eb-0392-4819-8114-e0c06d076ca6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629029 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629091 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629114 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629154 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629199 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629510 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629573 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629606 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629647 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.629898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.629865 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcf97eb-0392-4819-8114-e0c06d076ca6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.631252 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.631228 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.631551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.631511 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.650894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.650867 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcf97eb-0392-4819-8114-e0c06d076ca6-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.651094 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.651073 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v9td\" (UniqueName: \"kubernetes.io/projected/2bcf97eb-0392-4819-8114-e0c06d076ca6-kube-api-access-2v9td\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx\" (UID: \"2bcf97eb-0392-4819-8114-e0c06d076ca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:04.699411 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:04.699381 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:05.018403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.018375 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx"] Oct 29 23:08:05.019983 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:08:05.019955 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcf97eb_0392_4819_8114_e0c06d076ca6.slice/crio-28a83a2b38c5d31c75ae88c95c74e1b6fc07ec3b8223bc217dfa7ca69c15aba4 WatchSource:0}: Error finding container 28a83a2b38c5d31c75ae88c95c74e1b6fc07ec3b8223bc217dfa7ca69c15aba4: Status 404 returned error can't find the container with id 28a83a2b38c5d31c75ae88c95c74e1b6fc07ec3b8223bc217dfa7ca69c15aba4 Oct 29 23:08:05.022144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.022112 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:08:05.022229 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.022178 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:08:05.022229 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.022223 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:08:05.034945 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.034923 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt"] Oct 29 23:08:05.045827 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.045800 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh"] Oct 29 23:08:05.048340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.048318 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44907e0a-mxwbt"] Oct 29 23:08:05.050293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.050268 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc37b4-dvsmh"] Oct 29 23:08:05.221097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.221057 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc" path="/var/lib/kubelet/pods/7a18a25e-9a4a-4f3e-aeaf-eb79c0cdecfc/volumes" Oct 29 23:08:05.221729 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.221686 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d653f73e-726c-475d-b931-367ae7dfa5d8" path="/var/lib/kubelet/pods/d653f73e-726c-475d-b931-367ae7dfa5d8/volumes" Oct 29 23:08:05.288825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.288737 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" event={"ID":"2bcf97eb-0392-4819-8114-e0c06d076ca6","Type":"ContainerStarted","Data":"a5e06036bae3760dee55e8daf3b1a43fe8c56411528db1ed53d6f4b396061841"} Oct 29 23:08:05.288825 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.288803 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" event={"ID":"2bcf97eb-0392-4819-8114-e0c06d076ca6","Type":"ContainerStarted","Data":"28a83a2b38c5d31c75ae88c95c74e1b6fc07ec3b8223bc217dfa7ca69c15aba4"} Oct 29 23:08:05.339623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.339572 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podStartSLOduration=1.3395566190000001 podStartE2EDuration="1.339556619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:05.339012537 +0000 UTC m=+3152.233087459" watchObservedRunningTime="2026-10-29 23:08:05.339556619 +0000 UTC m=+3152.233631539" Oct 29 23:08:05.699941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:05.699904 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:06.702007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:06.701975 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.702426 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:06.702030 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 29 23:08:07.699856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:07.699825 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 29 23:08:07.700080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:07.699874 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 29 23:08:08.700127 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:08.700090 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 29 23:08:08.700528 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:08.700144 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 29 23:08:09.700735 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:09.700678 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 29 23:08:09.701145 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:09.700760 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 29 23:08:10.700085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:10.700046 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 29 23:08:10.700280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:10.700107 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 29 23:08:11.700148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:11.700109 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 29 23:08:11.700558 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:11.700168 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 29 23:08:13.228821 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:13.228770 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:13.229561 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:13.229538 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:13.230418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:13.230399 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" Oct 29 23:08:22.331123 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:08:22.331031 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:08:26.031226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:26.031192 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx"] Oct 29 23:08:26.044543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:26.044512 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d0ac72-qgkbx"] Oct 29 23:08:27.221412 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:27.221378 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57dcb70-3557-4506-9459-6aea71fe36e5" path="/var/lib/kubelet/pods/a57dcb70-3557-4506-9459-6aea71fe36e5/volumes" Oct 29 23:08:39.778952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:39.778917 5367 scope.go:122] "RemoveContainer" containerID="984c9e2b76f4c2cc39d9955b427be367faedb940edf99f30b160f24b6649bafd" Oct 29 23:08:39.788695 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:39.788671 5367 scope.go:122] "RemoveContainer" containerID="afec1ded198aacc40580a6c2a2fd5273265a1f51681efbbacbb4a575eb8714e0" Oct 29 23:08:39.797406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:39.797389 5367 scope.go:122] "RemoveContainer" containerID="79ec362f8fa6cb8234baf1f7b30ce21bf6e453a25f5399f905f04401b927c73d" Oct 29 23:08:39.806081 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:39.806060 5367 scope.go:122] "RemoveContainer" containerID="9c47710f22b91a462ab52c1d6e1cb78b5ec5a0449a6a35b968b74f7e70753a7f" Oct 29 23:08:46.826842 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.826805 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z"] Oct 29 23:08:46.831812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.831782 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:46.851075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.851048 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z"] Oct 29 23:08:46.861693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.861669 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:46.861876 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.861858 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:08:46.863077 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.863054 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:08:46.863194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.863059 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:08:46.974790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.974759 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxm6\" (UniqueName: \"kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:46.974969 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.974796 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:46.975048 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:46.975011 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.075858 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.075819 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.075858 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.075864 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxm6\" (UniqueName: \"kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.076140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.075885 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.076619 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.076597 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.078278 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.078224 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.097628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.097595 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxm6\" (UniqueName: \"kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6\") pod \"maas-api-revoke-sub-c1e271a9-mvh5z\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.144573 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.144544 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:47.441899 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.441874 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z"] Oct 29 23:08:47.443589 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:08:47.443560 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4127ef_9331_467e_855f_cd28e5f2db37.slice/crio-71359281a1a440e7c2d01b2f52ab38163456dd17ea8e4e3842ff0d5161d8705a WatchSource:0}: Error finding container 71359281a1a440e7c2d01b2f52ab38163456dd17ea8e4e3842ff0d5161d8705a: Status 404 returned error can't find the container with id 71359281a1a440e7c2d01b2f52ab38163456dd17ea8e4e3842ff0d5161d8705a Oct 29 23:08:47.580167 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:47.580131 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" event={"ID":"0a4127ef-9331-467e-855f-cd28e5f2db37","Type":"ContainerStarted","Data":"71359281a1a440e7c2d01b2f52ab38163456dd17ea8e4e3842ff0d5161d8705a"} Oct 29 23:08:48.588485 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:48.588452 5367 generic.go:356] "Generic (PLEG): container finished" podID="0a4127ef-9331-467e-855f-cd28e5f2db37" containerID="9d6d094cd63446fea5ac8a22c4ceecb71f56fea5eaac870f18d31398ca4acf1e" exitCode=0 Oct 29 23:08:48.588923 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:48.588530 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" event={"ID":"0a4127ef-9331-467e-855f-cd28e5f2db37","Type":"ContainerDied","Data":"9d6d094cd63446fea5ac8a22c4ceecb71f56fea5eaac870f18d31398ca4acf1e"} Oct 29 23:08:49.729270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.729245 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:49.808723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.803238 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6\" (UniqueName: \"kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6\") pod \"0a4127ef-9331-467e-855f-cd28e5f2db37\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " Oct 29 23:08:49.808723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.803339 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca\") pod \"0a4127ef-9331-467e-855f-cd28e5f2db37\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " Oct 29 23:08:49.808723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.803376 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca\") pod \"0a4127ef-9331-467e-855f-cd28e5f2db37\" (UID: \"0a4127ef-9331-467e-855f-cd28e5f2db37\") " Oct 29 23:08:49.808723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.804306 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca" pod "0a4127ef-9331-467e-855f-cd28e5f2db37" (UID: "0a4127ef-9331-467e-855f-cd28e5f2db37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:49.812743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.812689 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6" pod "0a4127ef-9331-467e-855f-cd28e5f2db37" (UID: "0a4127ef-9331-467e-855f-cd28e5f2db37"). InnerVolumeSpecName "kube-api-access-9nxm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:49.813138 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.813115 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca" pod "0a4127ef-9331-467e-855f-cd28e5f2db37" (UID: "0a4127ef-9331-467e-855f-cd28e5f2db37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:49.904480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.904408 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxm6\" (UniqueName: \"kubernetes.io/projected/0a4127ef-9331-467e-855f-cd28e5f2db37-kube-api-access-9nxm6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:08:49.904480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.904435 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:08:49.904480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:49.904445 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4127ef-9331-467e-855f-cd28e5f2db37-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:08:50.601715 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:50.601670 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" Oct 29 23:08:50.601901 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:50.601692 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z" event={"ID":"0a4127ef-9331-467e-855f-cd28e5f2db37","Type":"ContainerDied","Data":"71359281a1a440e7c2d01b2f52ab38163456dd17ea8e4e3842ff0d5161d8705a"} Oct 29 23:08:50.601901 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:08:50.601767 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71359281a1a440e7c2d01b2f52ab38163456dd17ea8e4e3842ff0d5161d8705a" Oct 29 23:08:52.329594 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:08:52.329551 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:09:07.776341 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.776308 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p"] Oct 29 23:09:07.776753 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.776715 5367 cpu_manager.go:414] "containerMap: removing container" podUID="0a4127ef-9331-467e-855f-cd28e5f2db37" containerName="revoke-keys" Oct 29 23:09:07.776753 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.776729 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4127ef-9331-467e-855f-cd28e5f2db37" containerName="revoke-keys" Oct 29 23:09:07.776847 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.776798 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4127ef-9331-467e-855f-cd28e5f2db37" containerName="revoke-keys" Oct 29 23:09:07.779100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.779079 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:07.811271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.811230 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:09:07.811462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.811301 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:09:07.811462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.811307 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:09:07.811462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.811350 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:09:07.813667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.813645 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p"] Oct 29 23:09:07.975538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.975507 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:07.975538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.975539 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drc2g\" (UniqueName: \"kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:07.975776 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:07.975566 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.076595 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.076558 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.076838 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.076598 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drc2g\" (UniqueName: \"kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.076838 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.076646 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.077467 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.077447 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.078953 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.078933 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.098630 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.098604 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drc2g\" (UniqueName: \"kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g\") pod \"maas-api-revoke-sub-2e1f5b5f-clw7p\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.391687 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.391608 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:08.689798 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.689765 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p"] Oct 29 23:09:08.691792 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:09:08.691758 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0e7199_f59f_4036_b778_3fa6facdca73.slice/crio-e418af4e8c527cfbdb39001e9f19a68bb2a6be7f154558b05b22f3abada6aa32 WatchSource:0}: Error finding container e418af4e8c527cfbdb39001e9f19a68bb2a6be7f154558b05b22f3abada6aa32: Status 404 returned error can't find the container with id e418af4e8c527cfbdb39001e9f19a68bb2a6be7f154558b05b22f3abada6aa32 Oct 29 23:09:08.743176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:08.743139 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" event={"ID":"5b0e7199-f59f-4036-b778-3fa6facdca73","Type":"ContainerStarted","Data":"e418af4e8c527cfbdb39001e9f19a68bb2a6be7f154558b05b22f3abada6aa32"} Oct 29 23:09:09.751228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:09.751133 5367 generic.go:356] "Generic (PLEG): container finished" podID="5b0e7199-f59f-4036-b778-3fa6facdca73" containerID="98cda1fce56cb126464eb450403dd7407d2452d79784b9e7e31850586097197c" exitCode=0 Oct 29 23:09:09.751228 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:09.751211 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" event={"ID":"5b0e7199-f59f-4036-b778-3fa6facdca73","Type":"ContainerDied","Data":"98cda1fce56cb126464eb450403dd7407d2452d79784b9e7e31850586097197c"} Oct 29 23:09:10.889999 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.889974 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:10.902450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.902426 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g\" (UniqueName: \"kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g\") pod \"5b0e7199-f59f-4036-b778-3fa6facdca73\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " Oct 29 23:09:10.902555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.902475 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca\") pod \"5b0e7199-f59f-4036-b778-3fa6facdca73\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " Oct 29 23:09:10.902555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.902538 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca\") pod \"5b0e7199-f59f-4036-b778-3fa6facdca73\" (UID: \"5b0e7199-f59f-4036-b778-3fa6facdca73\") " Oct 29 23:09:10.902923 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.902892 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca" pod "5b0e7199-f59f-4036-b778-3fa6facdca73" (UID: "5b0e7199-f59f-4036-b778-3fa6facdca73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:10.904547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.904518 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g" pod "5b0e7199-f59f-4036-b778-3fa6facdca73" (UID: "5b0e7199-f59f-4036-b778-3fa6facdca73"). InnerVolumeSpecName "kube-api-access-drc2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:10.904647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:10.904556 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca" pod "5b0e7199-f59f-4036-b778-3fa6facdca73" (UID: "5b0e7199-f59f-4036-b778-3fa6facdca73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:09:11.003937 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:11.003903 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:11.003937 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:11.003931 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drc2g\" (UniqueName: \"kubernetes.io/projected/5b0e7199-f59f-4036-b778-3fa6facdca73-kube-api-access-drc2g\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:11.003937 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:11.003943 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e7199-f59f-4036-b778-3fa6facdca73-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:11.765780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:11.765743 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" event={"ID":"5b0e7199-f59f-4036-b778-3fa6facdca73","Type":"ContainerDied","Data":"e418af4e8c527cfbdb39001e9f19a68bb2a6be7f154558b05b22f3abada6aa32"} Oct 29 23:09:11.765780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:11.765771 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p" Oct 29 23:09:11.766006 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:11.765802 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e418af4e8c527cfbdb39001e9f19a68bb2a6be7f154558b05b22f3abada6aa32" Oct 29 23:09:18.954164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.954123 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn"] Oct 29 23:09:18.954581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.954531 5367 cpu_manager.go:414] "containerMap: removing container" podUID="5b0e7199-f59f-4036-b778-3fa6facdca73" containerName="revoke-keys" Oct 29 23:09:18.954581 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.954546 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0e7199-f59f-4036-b778-3fa6facdca73" containerName="revoke-keys" Oct 29 23:09:18.954670 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.954637 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0e7199-f59f-4036-b778-3fa6facdca73" containerName="revoke-keys" Oct 29 23:09:18.957963 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.957941 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:18.991646 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.991600 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:09:18.991646 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.991636 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:09:18.991899 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.991810 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:09:18.991899 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.991863 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:09:18.994165 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:18.994142 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn"] Oct 29 23:09:19.077925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.077893 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qms7b\" (UniqueName: \"kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.078082 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.077951 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.078082 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.078056 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.179108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.179079 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qms7b\" (UniqueName: \"kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.179291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.179131 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.179291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.179164 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.179912 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.179887 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.181375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.181354 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.199747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.199686 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qms7b\" (UniqueName: \"kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b\") pod \"maas-api-revoke-sub-ec843018-7hfvn\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.275428 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.275356 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:19.575134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.575103 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn"] Oct 29 23:09:19.576276 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:09:19.576243 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63cc360_1b42_4226_9df0_fe1845ce30f7.slice/crio-ea9c00d7a743accf961bb6e1ea258c3452449e75a9f1ba425a6e5de4b6f684e3 WatchSource:0}: Error finding container ea9c00d7a743accf961bb6e1ea258c3452449e75a9f1ba425a6e5de4b6f684e3: Status 404 returned error can't find the container with id ea9c00d7a743accf961bb6e1ea258c3452449e75a9f1ba425a6e5de4b6f684e3 Oct 29 23:09:19.816826 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:19.816793 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" event={"ID":"c63cc360-1b42-4226-9df0-fe1845ce30f7","Type":"ContainerStarted","Data":"ea9c00d7a743accf961bb6e1ea258c3452449e75a9f1ba425a6e5de4b6f684e3"} Oct 29 23:09:20.824489 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:20.824458 5367 generic.go:356] "Generic (PLEG): container finished" podID="c63cc360-1b42-4226-9df0-fe1845ce30f7" containerID="6291748c031363d602288e8569c5a8cca160be28104bde43daf6a44ca5cdd3c6" exitCode=0 Oct 29 23:09:20.824894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:20.824494 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" event={"ID":"c63cc360-1b42-4226-9df0-fe1845ce30f7","Type":"ContainerDied","Data":"6291748c031363d602288e8569c5a8cca160be28104bde43daf6a44ca5cdd3c6"} Oct 29 23:09:21.963819 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:21.963794 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:22.006772 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.006746 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b\" (UniqueName: \"kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b\") pod \"c63cc360-1b42-4226-9df0-fe1845ce30f7\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " Oct 29 23:09:22.006920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.006780 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca\") pod \"c63cc360-1b42-4226-9df0-fe1845ce30f7\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " Oct 29 23:09:22.006920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.006815 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca\") pod \"c63cc360-1b42-4226-9df0-fe1845ce30f7\" (UID: \"c63cc360-1b42-4226-9df0-fe1845ce30f7\") " Oct 29 23:09:22.007170 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.007147 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca" pod "c63cc360-1b42-4226-9df0-fe1845ce30f7" (UID: "c63cc360-1b42-4226-9df0-fe1845ce30f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:22.009164 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.009133 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca" pod "c63cc360-1b42-4226-9df0-fe1845ce30f7" (UID: "c63cc360-1b42-4226-9df0-fe1845ce30f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:09:22.009275 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.009227 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b" pod "c63cc360-1b42-4226-9df0-fe1845ce30f7" (UID: "c63cc360-1b42-4226-9df0-fe1845ce30f7"). InnerVolumeSpecName "kube-api-access-qms7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:22.108584 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.108509 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qms7b\" (UniqueName: \"kubernetes.io/projected/c63cc360-1b42-4226-9df0-fe1845ce30f7-kube-api-access-qms7b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:22.108584 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.108535 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:22.108584 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.108546 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63cc360-1b42-4226-9df0-fe1845ce30f7-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:22.333577 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:09:22.333536 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:09:22.838382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.838330 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" Oct 29 23:09:22.838579 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.838388 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn" event={"ID":"c63cc360-1b42-4226-9df0-fe1845ce30f7","Type":"ContainerDied","Data":"ea9c00d7a743accf961bb6e1ea258c3452449e75a9f1ba425a6e5de4b6f684e3"} Oct 29 23:09:22.838579 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:22.838446 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9c00d7a743accf961bb6e1ea258c3452449e75a9f1ba425a6e5de4b6f684e3" Oct 29 23:09:39.845910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.845877 5367 scope.go:122] "RemoveContainer" containerID="66ab1e32e0ed50301a9b8eaf423dd716a97608aefcfa1a4b969e1809257d9e68" Oct 29 23:09:39.855503 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.855481 5367 scope.go:122] "RemoveContainer" containerID="8716665b5ec812c6614454d278ab3c390bb74b5879cdcdb4db109627d6dbaabe" Oct 29 23:09:39.864133 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.864114 5367 scope.go:122] "RemoveContainer" containerID="8ddb8fab79e09388eea322b30b91ebad99315a30886bb7907e398dd87664056c" Oct 29 23:09:39.872200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.872176 5367 scope.go:122] "RemoveContainer" containerID="e3d4de5f3314599033b635708fba17c8b775524acf777fe2219963bf6687ddd7" Oct 29 23:09:39.880258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.880242 5367 scope.go:122] "RemoveContainer" containerID="9a5e8a748e4094f29ab70743c5a1f1b04352af5c0c2d4f446e00804f0068ad6c" Oct 29 23:09:39.888334 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.888315 5367 scope.go:122] "RemoveContainer" containerID="797951fe3f57fd5bca8f37ececa432f3c701dc761a5cb36abd5f3d61a31f9b5d" Oct 29 23:09:39.896208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:39.896191 5367 scope.go:122] "RemoveContainer" containerID="28a00438ce31c7dbd75e247ad955d83e2cb5e681963a5c54c5d93e2b372c1b8a" Oct 29 23:09:46.763717 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.763616 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl"] Oct 29 23:09:46.764245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.764112 5367 cpu_manager.go:414] "containerMap: removing container" podUID="c63cc360-1b42-4226-9df0-fe1845ce30f7" containerName="revoke-keys" Oct 29 23:09:46.764245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.764131 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63cc360-1b42-4226-9df0-fe1845ce30f7" containerName="revoke-keys" Oct 29 23:09:46.764245 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.764220 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63cc360-1b42-4226-9df0-fe1845ce30f7" containerName="revoke-keys" Oct 29 23:09:46.769858 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.769838 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.787965 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.787924 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl"] Oct 29 23:09:46.797958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.797928 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:09:46.797958 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.797961 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:09:46.798205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.798060 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:09:46.799591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.799574 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:09:46.811668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.811638 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5w4\" (UniqueName: \"kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.811885 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.811769 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.811885 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.811811 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.912741 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.912692 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.912917 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.912751 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.912917 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.912779 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5w4\" (UniqueName: \"kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.913467 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.913438 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.915004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.914985 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:46.933959 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:46.933930 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5w4\" (UniqueName: \"kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4\") pod \"maas-api-revoke-sub-046e1343-xzgtl\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:47.083373 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:47.083339 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:47.387780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:47.387750 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl"] Oct 29 23:09:47.389239 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:09:47.389213 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab7be4f_bf30_4b5a_9d93_90d952b53565.slice/crio-9c6e35956b9512e1deb93f1908a2c0d284e4f1a61ca63f72ae6d41f68d6f73ce WatchSource:0}: Error finding container 9c6e35956b9512e1deb93f1908a2c0d284e4f1a61ca63f72ae6d41f68d6f73ce: Status 404 returned error can't find the container with id 9c6e35956b9512e1deb93f1908a2c0d284e4f1a61ca63f72ae6d41f68d6f73ce Oct 29 23:09:48.017487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:48.017387 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" event={"ID":"bab7be4f-bf30-4b5a-9d93-90d952b53565","Type":"ContainerStarted","Data":"9f59e4e1a993a936f807f71100c80fe7e76ffce9eaa8599cf9f3d50b470ead9a"} Oct 29 23:09:48.017487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:48.017475 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" event={"ID":"bab7be4f-bf30-4b5a-9d93-90d952b53565","Type":"ContainerStarted","Data":"9c6e35956b9512e1deb93f1908a2c0d284e4f1a61ca63f72ae6d41f68d6f73ce"} Oct 29 23:09:48.060772 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:48.060690 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" podStartSLOduration=2.060675355 podStartE2EDuration="2.060675355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:48.059366269 +0000 UTC m=+3254.953441192" watchObservedRunningTime="2026-10-29 23:09:48.060675355 +0000 UTC m=+3254.954750274" Oct 29 23:09:49.025615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:49.025581 5367 generic.go:356] "Generic (PLEG): container finished" podID="bab7be4f-bf30-4b5a-9d93-90d952b53565" containerID="9f59e4e1a993a936f807f71100c80fe7e76ffce9eaa8599cf9f3d50b470ead9a" exitCode=0 Oct 29 23:09:49.026011 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:49.025670 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" event={"ID":"bab7be4f-bf30-4b5a-9d93-90d952b53565","Type":"ContainerDied","Data":"9f59e4e1a993a936f807f71100c80fe7e76ffce9eaa8599cf9f3d50b470ead9a"} Oct 29 23:09:50.167936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.167909 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:50.244540 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.244514 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca\") pod \"bab7be4f-bf30-4b5a-9d93-90d952b53565\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " Oct 29 23:09:50.244691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.244549 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4\" (UniqueName: \"kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4\") pod \"bab7be4f-bf30-4b5a-9d93-90d952b53565\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " Oct 29 23:09:50.244691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.244591 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca\") pod \"bab7be4f-bf30-4b5a-9d93-90d952b53565\" (UID: \"bab7be4f-bf30-4b5a-9d93-90d952b53565\") " Oct 29 23:09:50.245025 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.245000 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca" pod "bab7be4f-bf30-4b5a-9d93-90d952b53565" (UID: "bab7be4f-bf30-4b5a-9d93-90d952b53565"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:50.246673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.246651 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca" pod "bab7be4f-bf30-4b5a-9d93-90d952b53565" (UID: "bab7be4f-bf30-4b5a-9d93-90d952b53565"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:09:50.246814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.246684 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4" pod "bab7be4f-bf30-4b5a-9d93-90d952b53565" (UID: "bab7be4f-bf30-4b5a-9d93-90d952b53565"). InnerVolumeSpecName "kube-api-access-4w5w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:50.346333 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.346306 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:50.346333 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.346333 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5w4\" (UniqueName: \"kubernetes.io/projected/bab7be4f-bf30-4b5a-9d93-90d952b53565-kube-api-access-4w5w4\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:50.346512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:50.346342 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab7be4f-bf30-4b5a-9d93-90d952b53565-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:09:51.040847 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:51.040813 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" event={"ID":"bab7be4f-bf30-4b5a-9d93-90d952b53565","Type":"ContainerDied","Data":"9c6e35956b9512e1deb93f1908a2c0d284e4f1a61ca63f72ae6d41f68d6f73ce"} Oct 29 23:09:51.041036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:51.040865 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c6e35956b9512e1deb93f1908a2c0d284e4f1a61ca63f72ae6d41f68d6f73ce" Oct 29 23:09:51.041036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:51.040824 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl" Oct 29 23:09:52.330035 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:09:52.330003 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:09:57.067161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.067124 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q"] Oct 29 23:09:57.067587 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.067571 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bab7be4f-bf30-4b5a-9d93-90d952b53565" containerName="revoke-keys" Oct 29 23:09:57.067587 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.067587 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab7be4f-bf30-4b5a-9d93-90d952b53565" containerName="revoke-keys" Oct 29 23:09:57.067728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.067694 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab7be4f-bf30-4b5a-9d93-90d952b53565" containerName="revoke-keys" Oct 29 23:09:57.071131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.071109 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.091493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.091464 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q"] Oct 29 23:09:57.101493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.101470 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:09:57.101656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.101531 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:09:57.106025 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.106003 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn629\" (UniqueName: \"kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.106153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.106049 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.106227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.106182 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.150680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.150655 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:09:57.150680 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.150679 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:09:57.207133 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.207092 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn629\" (UniqueName: \"kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.207283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.207148 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.207283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.207221 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.207820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.207802 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.209450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.209426 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.228134 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.228109 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn629\" (UniqueName: \"kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629\") pod \"maas-api-revoke-sub-d3fba056-7789q\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.384812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.384753 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:09:57.695771 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:09:57.695740 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494428d5_70e2_4013_8f56_29d416d542f1.slice/crio-5816307a5b464456cf10e046e482d5c06092d5cf7b5eef195b28e25e02d255de WatchSource:0}: Error finding container 5816307a5b464456cf10e046e482d5c06092d5cf7b5eef195b28e25e02d255de: Status 404 returned error can't find the container with id 5816307a5b464456cf10e046e482d5c06092d5cf7b5eef195b28e25e02d255de Oct 29 23:09:57.696139 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:57.696116 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q"] Oct 29 23:09:58.085176 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:58.085138 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" event={"ID":"494428d5-70e2-4013-8f56-29d416d542f1","Type":"ContainerStarted","Data":"2e4078ac2101d8685d0c1a80a460821c528e13b92cac3ac42c205b73f566aa3e"} Oct 29 23:09:58.085538 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:58.085201 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" event={"ID":"494428d5-70e2-4013-8f56-29d416d542f1","Type":"ContainerStarted","Data":"5816307a5b464456cf10e046e482d5c06092d5cf7b5eef195b28e25e02d255de"} Oct 29 23:09:58.150297 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:58.150244 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" podStartSLOduration=1.150226962 podStartE2EDuration="1.150226962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:58.148961685 +0000 UTC m=+3265.043036604" watchObservedRunningTime="2026-10-29 23:09:58.150226962 +0000 UTC m=+3265.044301881" Oct 29 23:09:59.095380 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:59.095343 5367 generic.go:356] "Generic (PLEG): container finished" podID="494428d5-70e2-4013-8f56-29d416d542f1" containerID="2e4078ac2101d8685d0c1a80a460821c528e13b92cac3ac42c205b73f566aa3e" exitCode=0 Oct 29 23:09:59.095830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:09:59.095439 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" event={"ID":"494428d5-70e2-4013-8f56-29d416d542f1","Type":"ContainerDied","Data":"2e4078ac2101d8685d0c1a80a460821c528e13b92cac3ac42c205b73f566aa3e"} Oct 29 23:10:00.537284 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.537256 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:10:00.643326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.643284 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca\") pod \"494428d5-70e2-4013-8f56-29d416d542f1\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " Oct 29 23:10:00.643541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.643388 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629\" (UniqueName: \"kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629\") pod \"494428d5-70e2-4013-8f56-29d416d542f1\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " Oct 29 23:10:00.643541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.643433 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca\") pod \"494428d5-70e2-4013-8f56-29d416d542f1\" (UID: \"494428d5-70e2-4013-8f56-29d416d542f1\") " Oct 29 23:10:00.643874 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.643837 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca" pod "494428d5-70e2-4013-8f56-29d416d542f1" (UID: "494428d5-70e2-4013-8f56-29d416d542f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:00.645729 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.645678 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629" pod "494428d5-70e2-4013-8f56-29d416d542f1" (UID: "494428d5-70e2-4013-8f56-29d416d542f1"). InnerVolumeSpecName "kube-api-access-mn629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:00.645831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.645777 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca" pod "494428d5-70e2-4013-8f56-29d416d542f1" (UID: "494428d5-70e2-4013-8f56-29d416d542f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:00.745331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.745244 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:00.745331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.745288 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494428d5-70e2-4013-8f56-29d416d542f1-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:00.745331 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:00.745305 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn629\" (UniqueName: \"kubernetes.io/projected/494428d5-70e2-4013-8f56-29d416d542f1-kube-api-access-mn629\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:01.143462 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:01.143429 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" Oct 29 23:10:01.143647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:01.143458 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q" event={"ID":"494428d5-70e2-4013-8f56-29d416d542f1","Type":"ContainerDied","Data":"5816307a5b464456cf10e046e482d5c06092d5cf7b5eef195b28e25e02d255de"} Oct 29 23:10:01.143647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:01.143541 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5816307a5b464456cf10e046e482d5c06092d5cf7b5eef195b28e25e02d255de" Oct 29 23:10:02.029513 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.029480 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv"] Oct 29 23:10:02.030068 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.029674 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" podUID="49bb90e7-c597-4a56-a31c-264a2387b1ca" containerName="discovery" containerID="cri-o://13bc1d41b2a1d4cd5e22f4bd3bce9814fb2fbd4863c5e7f4b28b8a1939a02c16" gracePeriod=30 Oct 29 23:10:02.152132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.152105 5367 generic.go:356] "Generic (PLEG): container finished" podID="49bb90e7-c597-4a56-a31c-264a2387b1ca" containerID="13bc1d41b2a1d4cd5e22f4bd3bce9814fb2fbd4863c5e7f4b28b8a1939a02c16" exitCode=0 Oct 29 23:10:02.152242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.152171 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" event={"ID":"49bb90e7-c597-4a56-a31c-264a2387b1ca","Type":"ContainerDied","Data":"13bc1d41b2a1d4cd5e22f4bd3bce9814fb2fbd4863c5e7f4b28b8a1939a02c16"} Oct 29 23:10:02.281886 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.281827 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 23:10:02.360506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.360469 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.360722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.360518 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.360722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.360562 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.360722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.360594 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.360722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.360649 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.360986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.360721 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.361056 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.361022 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:02.361056 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.361039 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token\") pod \"49bb90e7-c597-4a56-a31c-264a2387b1ca\" (UID: \"49bb90e7-c597-4a56-a31c-264a2387b1ca\") " Oct 29 23:10:02.361424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.361390 5367 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-ca-configmap\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:02.363046 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.363020 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:02.363179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.363104 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:02.363358 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.363334 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:02.363452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.363366 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "kube-api-access-sqgfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:02.363452 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.363438 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:02.363572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.363549 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs" pod "49bb90e7-c597-4a56-a31c-264a2387b1ca" (UID: "49bb90e7-c597-4a56-a31c-264a2387b1ca"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:02.462688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.462651 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgfl\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-kube-api-access-sqgfl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:02.462688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.462690 5367 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb90e7-c597-4a56-a31c-264a2387b1ca-local-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:02.462924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.462720 5367 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-cacerts\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:02.462924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.462733 5367 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-csr-dns-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:02.462924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.462747 5367 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-kubeconfig\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:02.462924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:02.462759 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb90e7-c597-4a56-a31c-264a2387b1ca-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:03.159353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:03.159319 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" Oct 29 23:10:03.159353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:03.159347 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv" event={"ID":"49bb90e7-c597-4a56-a31c-264a2387b1ca","Type":"ContainerDied","Data":"2ab285f599bf4ad79453df5055775010f33fda5265fe2689a6c6399ca4b67c60"} Oct 29 23:10:03.159986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:03.159412 5367 scope.go:122] "RemoveContainer" containerID="13bc1d41b2a1d4cd5e22f4bd3bce9814fb2fbd4863c5e7f4b28b8a1939a02c16" Oct 29 23:10:03.233204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:03.233174 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv"] Oct 29 23:10:03.246910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:03.246883 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xc4fv"] Oct 29 23:10:05.219428 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:05.219393 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49bb90e7-c597-4a56-a31c-264a2387b1ca" path="/var/lib/kubelet/pods/49bb90e7-c597-4a56-a31c-264a2387b1ca/volumes" Oct 29 23:10:18.544023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.543988 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc"] Oct 29 23:10:18.544607 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.544582 5367 cpu_manager.go:414] "containerMap: removing container" podUID="494428d5-70e2-4013-8f56-29d416d542f1" containerName="revoke-keys" Oct 29 23:10:18.544671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.544612 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494428d5-70e2-4013-8f56-29d416d542f1" containerName="revoke-keys" Oct 29 23:10:18.544671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.544631 5367 cpu_manager.go:414] "containerMap: removing container" podUID="49bb90e7-c597-4a56-a31c-264a2387b1ca" containerName="discovery" Oct 29 23:10:18.544671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.544640 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49bb90e7-c597-4a56-a31c-264a2387b1ca" containerName="discovery" Oct 29 23:10:18.544855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.544768 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="494428d5-70e2-4013-8f56-29d416d542f1" containerName="revoke-keys" Oct 29 23:10:18.544855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.544781 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="49bb90e7-c597-4a56-a31c-264a2387b1ca" containerName="discovery" Oct 29 23:10:18.548693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.548676 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.567039 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.567005 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc"] Oct 29 23:10:18.576728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.576691 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:18.576875 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.576854 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:10:18.576934 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.576912 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:10:18.578618 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.578598 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:10:18.663887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.663854 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.664062 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.663895 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85tzl\" (UniqueName: \"kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.664062 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.664039 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.764924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.764893 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.765088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.764944 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.765088 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.765065 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tzl\" (UniqueName: \"kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.765501 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.765484 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.767224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.767208 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.786234 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.786215 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tzl\" (UniqueName: \"kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl\") pod \"maas-api-revoke-sub-ae594979-482tc\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:18.867808 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:18.867783 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:19.163330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:19.163300 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc"] Oct 29 23:10:19.165411 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:10:19.165378 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f189a1_e108_4ce4_86a5_416976ad4961.slice/crio-3773383aa7eae28f443de477d88d8cedbd863a458ae5e5fe64feec45dabc1e7a WatchSource:0}: Error finding container 3773383aa7eae28f443de477d88d8cedbd863a458ae5e5fe64feec45dabc1e7a: Status 404 returned error can't find the container with id 3773383aa7eae28f443de477d88d8cedbd863a458ae5e5fe64feec45dabc1e7a Oct 29 23:10:19.263898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:19.263862 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" event={"ID":"85f189a1-e108-4ce4-86a5-416976ad4961","Type":"ContainerStarted","Data":"3773383aa7eae28f443de477d88d8cedbd863a458ae5e5fe64feec45dabc1e7a"} Oct 29 23:10:20.271534 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:20.271442 5367 generic.go:356] "Generic (PLEG): container finished" podID="85f189a1-e108-4ce4-86a5-416976ad4961" containerID="2489ecd19c83c4d106c375e2a92445f97ec0e07ae04a93cbf14202db89b883cf" exitCode=0 Oct 29 23:10:20.271998 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:20.271531 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" event={"ID":"85f189a1-e108-4ce4-86a5-416976ad4961","Type":"ContainerDied","Data":"2489ecd19c83c4d106c375e2a92445f97ec0e07ae04a93cbf14202db89b883cf"} Oct 29 23:10:21.409918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.409893 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:21.590277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.590242 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca\") pod \"85f189a1-e108-4ce4-86a5-416976ad4961\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " Oct 29 23:10:21.590507 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.590356 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl\" (UniqueName: \"kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl\") pod \"85f189a1-e108-4ce4-86a5-416976ad4961\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " Oct 29 23:10:21.590507 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.590386 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca\") pod \"85f189a1-e108-4ce4-86a5-416976ad4961\" (UID: \"85f189a1-e108-4ce4-86a5-416976ad4961\") " Oct 29 23:10:21.590815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.590790 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca" pod "85f189a1-e108-4ce4-86a5-416976ad4961" (UID: "85f189a1-e108-4ce4-86a5-416976ad4961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:21.592517 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.592485 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca" pod "85f189a1-e108-4ce4-86a5-416976ad4961" (UID: "85f189a1-e108-4ce4-86a5-416976ad4961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:21.592517 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.592501 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl" pod "85f189a1-e108-4ce4-86a5-416976ad4961" (UID: "85f189a1-e108-4ce4-86a5-416976ad4961"). InnerVolumeSpecName "kube-api-access-85tzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:21.691895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.691864 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85tzl\" (UniqueName: \"kubernetes.io/projected/85f189a1-e108-4ce4-86a5-416976ad4961-kube-api-access-85tzl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:21.691895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.691889 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:21.691895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:21.691899 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f189a1-e108-4ce4-86a5-416976ad4961-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:22.283883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:22.283851 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" Oct 29 23:10:22.284071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:22.283891 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc" event={"ID":"85f189a1-e108-4ce4-86a5-416976ad4961","Type":"ContainerDied","Data":"3773383aa7eae28f443de477d88d8cedbd863a458ae5e5fe64feec45dabc1e7a"} Oct 29 23:10:22.284071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:22.283939 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3773383aa7eae28f443de477d88d8cedbd863a458ae5e5fe64feec45dabc1e7a" Oct 29 23:10:22.329887 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:10:22.329853 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:10:30.031494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:30.031462 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5"] Oct 29 23:10:30.047290 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:30.047253 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk"] Oct 29 23:10:30.050349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:30.050329 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb61c20a-sxxt5"] Oct 29 23:10:30.052356 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:30.052337 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901d3578-zkcrk"] Oct 29 23:10:31.219714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:31.219656 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852c31db-fafa-4c58-8b6e-c89034730fc6" path="/var/lib/kubelet/pods/852c31db-fafa-4c58-8b6e-c89034730fc6/volumes" Oct 29 23:10:31.220137 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:31.220122 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c4e342-2995-49f9-b478-4d93a4a012d4" path="/var/lib/kubelet/pods/c3c4e342-2995-49f9-b478-4d93a4a012d4/volumes" Oct 29 23:10:39.941138 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:39.941101 5367 scope.go:122] "RemoveContainer" containerID="5c49eb5156bfea56d8a36eba792bd7156f2027db722e9c422ce8cfa9e7748b89" Oct 29 23:10:39.950317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:39.950293 5367 scope.go:122] "RemoveContainer" containerID="24c9aea470c56e231e1b4735ada66a67cc96fb35833662e2083b6ca53674231b" Oct 29 23:10:39.994025 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:39.993998 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8"] Oct 29 23:10:39.994445 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:39.994421 5367 cpu_manager.go:414] "containerMap: removing container" podUID="85f189a1-e108-4ce4-86a5-416976ad4961" containerName="revoke-keys" Oct 29 23:10:39.994560 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:39.994446 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f189a1-e108-4ce4-86a5-416976ad4961" containerName="revoke-keys" Oct 29 23:10:39.994642 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:39.994627 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f189a1-e108-4ce4-86a5-416976ad4961" containerName="revoke-keys" Oct 29 23:10:40.002830 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.000469 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.017270 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.017242 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8"] Oct 29 23:10:40.028035 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.028002 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:10:40.028202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.028046 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:10:40.028202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.028063 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:10:40.030925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.030906 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:40.095383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.095341 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.095383 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.095380 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.095617 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.095411 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshzk\" (UniqueName: \"kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.197104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.197015 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.197104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.197059 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.197104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.197094 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshzk\" (UniqueName: \"kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.197811 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.197791 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.199424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.199403 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.218660 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.218632 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshzk\" (UniqueName: \"kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk\") pod \"maas-api-revoke-sub-a4aef7d7-j7nb8\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.315900 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.315867 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:40.613849 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:40.613824 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8"] Oct 29 23:10:40.615354 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:10:40.615325 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb7dddf_aef0_4a68_a899_14ad6fbe70d7.slice/crio-fcd2c6646d5d37b6cf0562650aa6d728174b565f7ed594aa074df54810c97c56 WatchSource:0}: Error finding container fcd2c6646d5d37b6cf0562650aa6d728174b565f7ed594aa074df54810c97c56: Status 404 returned error can't find the container with id fcd2c6646d5d37b6cf0562650aa6d728174b565f7ed594aa074df54810c97c56 Oct 29 23:10:41.433198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:41.433164 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" event={"ID":"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7","Type":"ContainerStarted","Data":"0f021c693335188a150da001f17cd290e1b14ff2676641615446356105d946bf"} Oct 29 23:10:41.433607 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:41.433214 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" event={"ID":"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7","Type":"ContainerStarted","Data":"fcd2c6646d5d37b6cf0562650aa6d728174b565f7ed594aa074df54810c97c56"} Oct 29 23:10:41.477647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:41.477586 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" podStartSLOduration=2.477571867 podStartE2EDuration="2.477571867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:41.474732715 +0000 UTC m=+3308.368807633" watchObservedRunningTime="2026-10-29 23:10:41.477571867 +0000 UTC m=+3308.371646786" Oct 29 23:10:42.441816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:42.441783 5367 generic.go:356] "Generic (PLEG): container finished" podID="3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" containerID="0f021c693335188a150da001f17cd290e1b14ff2676641615446356105d946bf" exitCode=0 Oct 29 23:10:42.442243 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:42.441877 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" event={"ID":"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7","Type":"ContainerDied","Data":"0f021c693335188a150da001f17cd290e1b14ff2676641615446356105d946bf"} Oct 29 23:10:43.582160 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.582132 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:43.632275 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.632242 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca\") pod \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " Oct 29 23:10:43.632458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.632308 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca\") pod \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " Oct 29 23:10:43.632458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.632404 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk\" (UniqueName: \"kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk\") pod \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\" (UID: \"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7\") " Oct 29 23:10:43.632589 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.632530 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca" pod "3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" (UID: "3fb7dddf-aef0-4a68-a899-14ad6fbe70d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:43.632733 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.632712 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:43.634351 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.634321 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca" pod "3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" (UID: "3fb7dddf-aef0-4a68-a899-14ad6fbe70d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:43.634487 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.634435 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk" pod "3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" (UID: "3fb7dddf-aef0-4a68-a899-14ad6fbe70d7"). InnerVolumeSpecName "kube-api-access-lshzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:43.733624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.733545 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:43.733624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:43.733574 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lshzk\" (UniqueName: \"kubernetes.io/projected/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7-kube-api-access-lshzk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:10:44.455080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:44.455041 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" event={"ID":"3fb7dddf-aef0-4a68-a899-14ad6fbe70d7","Type":"ContainerDied","Data":"fcd2c6646d5d37b6cf0562650aa6d728174b565f7ed594aa074df54810c97c56"} Oct 29 23:10:44.455080 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:44.455059 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8" Oct 29 23:10:44.455304 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:10:44.455090 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcd2c6646d5d37b6cf0562650aa6d728174b565f7ed594aa074df54810c97c56" Oct 29 23:10:52.330391 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:10:52.330352 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:11:01.099395 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.099354 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht"] Oct 29 23:11:01.099831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.099782 5367 cpu_manager.go:414] "containerMap: removing container" podUID="3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" containerName="revoke-keys" Oct 29 23:11:01.099831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.099795 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" containerName="revoke-keys" Oct 29 23:11:01.099913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.099865 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" containerName="revoke-keys" Oct 29 23:11:01.103253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.103234 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.124158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.124123 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht"] Oct 29 23:11:01.134097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.134067 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:01.134311 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.134294 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:11:01.134410 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.134389 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:11:01.134484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.134388 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:11:01.233138 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.233109 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.233336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.233143 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.233336 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.233189 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mnvv\" (UniqueName: \"kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.334525 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.334487 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mnvv\" (UniqueName: \"kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.334684 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.334599 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.334684 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.334617 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.335407 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.335385 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.337018 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.336999 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.355856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.355801 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mnvv\" (UniqueName: \"kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv\") pod \"maas-api-revoke-sub-79805ca9-8trht\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.416845 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.416809 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:01.715791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:01.715763 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht"] Oct 29 23:11:01.717195 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:11:01.717161 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ade951_6c15_460c_b543_7a43cb43aabc.slice/crio-2de8d4d87e22f65e1b32ba0536fb2f268435cb67b0ca73cc7755adcaad8a07cc WatchSource:0}: Error finding container 2de8d4d87e22f65e1b32ba0536fb2f268435cb67b0ca73cc7755adcaad8a07cc: Status 404 returned error can't find the container with id 2de8d4d87e22f65e1b32ba0536fb2f268435cb67b0ca73cc7755adcaad8a07cc Oct 29 23:11:02.574421 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:02.574388 5367 generic.go:356] "Generic (PLEG): container finished" podID="91ade951-6c15-460c-b543-7a43cb43aabc" containerID="78ba611a75a547938db073c8357fdd857db1d745f28a021f46cefb1f2f3999d6" exitCode=0 Oct 29 23:11:02.574891 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:02.574489 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" event={"ID":"91ade951-6c15-460c-b543-7a43cb43aabc","Type":"ContainerDied","Data":"78ba611a75a547938db073c8357fdd857db1d745f28a021f46cefb1f2f3999d6"} Oct 29 23:11:02.574891 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:02.574541 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" event={"ID":"91ade951-6c15-460c-b543-7a43cb43aabc","Type":"ContainerStarted","Data":"2de8d4d87e22f65e1b32ba0536fb2f268435cb67b0ca73cc7755adcaad8a07cc"} Oct 29 23:11:03.707514 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.707486 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:03.860979 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.860899 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv\" (UniqueName: \"kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv\") pod \"91ade951-6c15-460c-b543-7a43cb43aabc\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " Oct 29 23:11:03.861161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.860985 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca\") pod \"91ade951-6c15-460c-b543-7a43cb43aabc\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " Oct 29 23:11:03.861161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.861036 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca\") pod \"91ade951-6c15-460c-b543-7a43cb43aabc\" (UID: \"91ade951-6c15-460c-b543-7a43cb43aabc\") " Oct 29 23:11:03.861354 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.861319 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca" pod "91ade951-6c15-460c-b543-7a43cb43aabc" (UID: "91ade951-6c15-460c-b543-7a43cb43aabc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:03.863089 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.863066 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca" pod "91ade951-6c15-460c-b543-7a43cb43aabc" (UID: "91ade951-6c15-460c-b543-7a43cb43aabc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:03.863189 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.863171 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv" pod "91ade951-6c15-460c-b543-7a43cb43aabc" (UID: "91ade951-6c15-460c-b543-7a43cb43aabc"). InnerVolumeSpecName "kube-api-access-5mnvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:03.962325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.962293 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mnvv\" (UniqueName: \"kubernetes.io/projected/91ade951-6c15-460c-b543-7a43cb43aabc-kube-api-access-5mnvv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.962325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.962319 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.962325 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:03.962329 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ade951-6c15-460c-b543-7a43cb43aabc-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:04.587799 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:04.587765 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" event={"ID":"91ade951-6c15-460c-b543-7a43cb43aabc","Type":"ContainerDied","Data":"2de8d4d87e22f65e1b32ba0536fb2f268435cb67b0ca73cc7755adcaad8a07cc"} Oct 29 23:11:04.587799 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:04.587792 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht" Oct 29 23:11:04.588037 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:04.587812 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de8d4d87e22f65e1b32ba0536fb2f268435cb67b0ca73cc7755adcaad8a07cc" Oct 29 23:11:18.992320 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:18.992236 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm"] Oct 29 23:11:18.992801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:18.992604 5367 cpu_manager.go:414] "containerMap: removing container" podUID="91ade951-6c15-460c-b543-7a43cb43aabc" containerName="revoke-keys" Oct 29 23:11:18.992801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:18.992615 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ade951-6c15-460c-b543-7a43cb43aabc" containerName="revoke-keys" Oct 29 23:11:18.992801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:18.992683 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ade951-6c15-460c-b543-7a43cb43aabc" containerName="revoke-keys" Oct 29 23:11:18.995970 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:18.995947 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.014956 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.014931 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm"] Oct 29 23:11:19.025142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.025106 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:11:19.025298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.025228 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:11:19.026714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.026687 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:19.026861 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.026847 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:11:19.051554 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.051515 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.051782 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.051756 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.051867 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.051821 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vkm\" (UniqueName: \"kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.152623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.152587 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.152847 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.152631 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vkm\" (UniqueName: \"kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.152847 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.152672 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.153375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.153354 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.154928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.154905 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.173438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.173416 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vkm\" (UniqueName: \"kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm\") pod \"maas-api-revoke-sub-0b7e8edb-fjmjm\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.309049 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.309019 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:19.604796 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.604771 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm"] Oct 29 23:11:19.605857 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:11:19.605831 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf149d55_ae3f_4c95_b720_4f6b1b74f7e5.slice/crio-5641ab9c83c12f08d8335765dc963fd9b4ed5e6f198d04ff4ee3e95ec85e833b WatchSource:0}: Error finding container 5641ab9c83c12f08d8335765dc963fd9b4ed5e6f198d04ff4ee3e95ec85e833b: Status 404 returned error can't find the container with id 5641ab9c83c12f08d8335765dc963fd9b4ed5e6f198d04ff4ee3e95ec85e833b Oct 29 23:11:19.689406 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:19.689370 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" event={"ID":"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5","Type":"ContainerStarted","Data":"5641ab9c83c12f08d8335765dc963fd9b4ed5e6f198d04ff4ee3e95ec85e833b"} Oct 29 23:11:20.696523 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:20.696491 5367 generic.go:356] "Generic (PLEG): container finished" podID="bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" containerID="b99dcb4a62990823c49fd211a5ce68eb82c9acb313764734195e1355d9fe809e" exitCode=0 Oct 29 23:11:20.696927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:20.696570 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" event={"ID":"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5","Type":"ContainerDied","Data":"b99dcb4a62990823c49fd211a5ce68eb82c9acb313764734195e1355d9fe809e"} Oct 29 23:11:21.841832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.841805 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:21.878532 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.878496 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca\") pod \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " Oct 29 23:11:21.878749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.878570 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm\" (UniqueName: \"kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm\") pod \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " Oct 29 23:11:21.878749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.878675 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca\") pod \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\" (UID: \"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5\") " Oct 29 23:11:21.879026 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.878997 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca" pod "bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" (UID: "bf149d55-ae3f-4c95-b720-4f6b1b74f7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:21.880661 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.880631 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca" pod "bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" (UID: "bf149d55-ae3f-4c95-b720-4f6b1b74f7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:21.880770 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.880716 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm" pod "bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" (UID: "bf149d55-ae3f-4c95-b720-4f6b1b74f7e5"). InnerVolumeSpecName "kube-api-access-w7vkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:21.980222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.980137 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:21.980222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.980172 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vkm\" (UniqueName: \"kubernetes.io/projected/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-kube-api-access-w7vkm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:21.980222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:21.980188 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:22.331128 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:11:22.331087 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:11:22.712615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:22.712528 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" Oct 29 23:11:22.712615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:22.712563 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm" event={"ID":"bf149d55-ae3f-4c95-b720-4f6b1b74f7e5","Type":"ContainerDied","Data":"5641ab9c83c12f08d8335765dc963fd9b4ed5e6f198d04ff4ee3e95ec85e833b"} Oct 29 23:11:22.712851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:22.712618 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5641ab9c83c12f08d8335765dc963fd9b4ed5e6f198d04ff4ee3e95ec85e833b" Oct 29 23:11:29.720933 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.720890 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9"] Oct 29 23:11:29.722293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.722264 5367 cpu_manager.go:414] "containerMap: removing container" podUID="bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" containerName="revoke-keys" Oct 29 23:11:29.722473 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.722457 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" containerName="revoke-keys" Oct 29 23:11:29.722852 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.722835 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" containerName="revoke-keys" Oct 29 23:11:29.727875 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.727853 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.741763 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.741732 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9"] Oct 29 23:11:29.751319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.751283 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:29.751319 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.751288 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:11:29.757771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.757744 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:11:29.757771 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.757757 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:11:29.853112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.853078 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vpl\" (UniqueName: \"kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.853112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.853111 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.853347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.853236 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.954128 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.954089 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.954340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.954154 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vpl\" (UniqueName: \"kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.954340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.954184 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.954961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.954937 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.956583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.956557 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:29.975329 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:29.975273 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vpl\" (UniqueName: \"kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl\") pod \"maas-api-revoke-sub-3d4156f1-plnd9\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:30.041612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:30.041583 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:30.341472 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:30.341445 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9"] Oct 29 23:11:30.343000 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:11:30.342972 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d389c43_09e2_466d_add3_7a2d3ecee7ab.slice/crio-0020bf02441b333232316b1ad98ee115d27af7004aa8fb15b30c595e5aab741e WatchSource:0}: Error finding container 0020bf02441b333232316b1ad98ee115d27af7004aa8fb15b30c595e5aab741e: Status 404 returned error can't find the container with id 0020bf02441b333232316b1ad98ee115d27af7004aa8fb15b30c595e5aab741e Oct 29 23:11:30.764673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:30.764636 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" event={"ID":"2d389c43-09e2-466d-add3-7a2d3ecee7ab","Type":"ContainerStarted","Data":"bfcb315f9cc2ef8b530bc71b8e92a553a8b9c24a6d034d737fcd4fb38d206fbc"} Oct 29 23:11:30.765100 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:30.764723 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" event={"ID":"2d389c43-09e2-466d-add3-7a2d3ecee7ab","Type":"ContainerStarted","Data":"0020bf02441b333232316b1ad98ee115d27af7004aa8fb15b30c595e5aab741e"} Oct 29 23:11:30.810223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:30.810169 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" podStartSLOduration=1.8101554549999999 podStartE2EDuration="1.810155455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:30.807628902 +0000 UTC m=+3357.701703823" watchObservedRunningTime="2026-10-29 23:11:30.810155455 +0000 UTC m=+3357.704230375" Oct 29 23:11:31.772693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:31.772652 5367 generic.go:356] "Generic (PLEG): container finished" podID="2d389c43-09e2-466d-add3-7a2d3ecee7ab" containerID="bfcb315f9cc2ef8b530bc71b8e92a553a8b9c24a6d034d737fcd4fb38d206fbc" exitCode=0 Oct 29 23:11:31.773155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:31.772738 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" event={"ID":"2d389c43-09e2-466d-add3-7a2d3ecee7ab","Type":"ContainerDied","Data":"bfcb315f9cc2ef8b530bc71b8e92a553a8b9c24a6d034d737fcd4fb38d206fbc"} Oct 29 23:11:32.907788 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.907762 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:32.981309 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.981275 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl\" (UniqueName: \"kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl\") pod \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " Oct 29 23:11:32.981475 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.981397 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca\") pod \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " Oct 29 23:11:32.981531 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.981514 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca\") pod \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\" (UID: \"2d389c43-09e2-466d-add3-7a2d3ecee7ab\") " Oct 29 23:11:32.981878 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.981851 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca" pod "2d389c43-09e2-466d-add3-7a2d3ecee7ab" (UID: "2d389c43-09e2-466d-add3-7a2d3ecee7ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:32.983398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.983377 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl" pod "2d389c43-09e2-466d-add3-7a2d3ecee7ab" (UID: "2d389c43-09e2-466d-add3-7a2d3ecee7ab"). InnerVolumeSpecName "kube-api-access-h4vpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:32.983495 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:32.983465 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca" pod "2d389c43-09e2-466d-add3-7a2d3ecee7ab" (UID: "2d389c43-09e2-466d-add3-7a2d3ecee7ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:33.082491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:33.082458 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:33.082491 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:33.082489 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d389c43-09e2-466d-add3-7a2d3ecee7ab-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:33.082694 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:33.082498 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vpl\" (UniqueName: \"kubernetes.io/projected/2d389c43-09e2-466d-add3-7a2d3ecee7ab-kube-api-access-h4vpl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:33.786579 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:33.786540 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" event={"ID":"2d389c43-09e2-466d-add3-7a2d3ecee7ab","Type":"ContainerDied","Data":"0020bf02441b333232316b1ad98ee115d27af7004aa8fb15b30c595e5aab741e"} Oct 29 23:11:33.786794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:33.786598 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0020bf02441b333232316b1ad98ee115d27af7004aa8fb15b30c595e5aab741e" Oct 29 23:11:33.786794 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:33.786551 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9" Oct 29 23:11:39.993079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:39.993043 5367 scope.go:122] "RemoveContainer" containerID="5852389f96f1e0faf461a4615144e31b79d7e2d9227bbd1961bfa53809f05bdf" Oct 29 23:11:40.002973 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.002946 5367 scope.go:122] "RemoveContainer" containerID="f5b26e88e9f6ab7e64c5ce1fe97dc23c85dd9b95300e76028539b32c30acfc7c" Oct 29 23:11:40.011524 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.011507 5367 scope.go:122] "RemoveContainer" containerID="1b4e7a33c8fc679fea7e291716c204fed2cbe1f885ccd2617cbad8da9dec6ff2" Oct 29 23:11:40.019692 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.019660 5367 scope.go:122] "RemoveContainer" containerID="e3cd8ace1628e33c4bd25ebe569383a8be3a7018331ff6304dbf3e25c550e457" Oct 29 23:11:40.028179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.028159 5367 scope.go:122] "RemoveContainer" containerID="67c8bd646e3e3c7b5b275aba4b0a060214aea22058f5e2b1ae9b5bce32d62f2d" Oct 29 23:11:40.036404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.036386 5367 scope.go:122] "RemoveContainer" containerID="18ee0d3d79d3ac9d40547c6fc182f62e2745b0e2b3a84b6a575d363ebea9d094" Oct 29 23:11:40.044878 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.044858 5367 scope.go:122] "RemoveContainer" containerID="2740a6eca586da1f0110dffdd93261203b98b482f60fb56b89eca5184abf0d55" Oct 29 23:11:40.052339 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:40.052321 5367 scope.go:122] "RemoveContainer" containerID="94a8024f8ad997263ab790e1833f9af045568776b3dd972ce38cf9a638aa028c" Oct 29 23:11:42.035994 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:42.035953 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg"] Oct 29 23:11:42.050008 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:42.049969 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae1a01-z5wsg"] Oct 29 23:11:43.243192 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:43.243151 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4423cd5c-a24c-4a2f-a6c1-c08af04c852e" path="/var/lib/kubelet/pods/4423cd5c-a24c-4a2f-a6c1-c08af04c852e/volumes" Oct 29 23:11:51.008690 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.008655 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9"] Oct 29 23:11:51.022151 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.022114 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-plnd9"] Oct 29 23:11:51.051933 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.051898 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn"] Oct 29 23:11:51.052316 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.052303 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2d389c43-09e2-466d-add3-7a2d3ecee7ab" containerName="revoke-keys" Oct 29 23:11:51.052372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.052318 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d389c43-09e2-466d-add3-7a2d3ecee7ab" containerName="revoke-keys" Oct 29 23:11:51.052415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.052381 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d389c43-09e2-466d-add3-7a2d3ecee7ab" containerName="revoke-keys" Oct 29 23:11:51.057183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.057165 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.073803 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.073779 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:51.074486 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.074447 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:11:51.074612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.074514 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:11:51.074612 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.074545 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:11:51.075740 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.075717 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn"] Oct 29 23:11:51.101124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.101096 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.101303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.101206 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.101303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.101232 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v6b\" (UniqueName: \"kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.202280 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.202245 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.202463 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.202309 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.202463 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.202327 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v6b\" (UniqueName: \"kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.203054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.203026 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.204656 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.204631 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.219455 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.219424 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d389c43-09e2-466d-add3-7a2d3ecee7ab" path="/var/lib/kubelet/pods/2d389c43-09e2-466d-add3-7a2d3ecee7ab/volumes" Oct 29 23:11:51.222665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.222644 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v6b\" (UniqueName: \"kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b\") pod \"maas-api-revoke-sub-3d4156f1-qlswn\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.369851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.369818 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:51.675172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.675140 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn"] Oct 29 23:11:51.676308 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:11:51.676282 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb73f5eb_fa9b_4002_aa96_fed7cc57a472.slice/crio-f5b1194c00e6a325852d56f4c88e21d89d7e4f04840bb646fd321ff9011f1377 WatchSource:0}: Error finding container f5b1194c00e6a325852d56f4c88e21d89d7e4f04840bb646fd321ff9011f1377: Status 404 returned error can't find the container with id f5b1194c00e6a325852d56f4c88e21d89d7e4f04840bb646fd321ff9011f1377 Oct 29 23:11:51.897429 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.897394 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" event={"ID":"db73f5eb-fa9b-4002-aa96-fed7cc57a472","Type":"ContainerStarted","Data":"2558a49a7c037cebf1355a5ef3bdd495e1d807ead090a0e362a47449ff001f5b"} Oct 29 23:11:51.897603 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.897450 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" event={"ID":"db73f5eb-fa9b-4002-aa96-fed7cc57a472","Type":"ContainerStarted","Data":"f5b1194c00e6a325852d56f4c88e21d89d7e4f04840bb646fd321ff9011f1377"} Oct 29 23:11:51.942183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:51.942083 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" podStartSLOduration=0.942067486 podStartE2EDuration="942.067486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:51.940032909 +0000 UTC m=+3378.834107844" watchObservedRunningTime="2026-10-29 23:11:51.942067486 +0000 UTC m=+3378.836142407" Oct 29 23:11:52.331254 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:11:52.331210 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:11:52.904863 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:52.904831 5367 generic.go:356] "Generic (PLEG): container finished" podID="db73f5eb-fa9b-4002-aa96-fed7cc57a472" containerID="2558a49a7c037cebf1355a5ef3bdd495e1d807ead090a0e362a47449ff001f5b" exitCode=0 Oct 29 23:11:52.905052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:52.904902 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" event={"ID":"db73f5eb-fa9b-4002-aa96-fed7cc57a472","Type":"ContainerDied","Data":"2558a49a7c037cebf1355a5ef3bdd495e1d807ead090a0e362a47449ff001f5b"} Oct 29 23:11:54.040206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.040176 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:54.131609 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.131571 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b\" (UniqueName: \"kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b\") pod \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " Oct 29 23:11:54.131812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.131630 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca\") pod \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " Oct 29 23:11:54.131812 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.131670 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca\") pod \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\" (UID: \"db73f5eb-fa9b-4002-aa96-fed7cc57a472\") " Oct 29 23:11:54.132010 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.131988 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca" pod "db73f5eb-fa9b-4002-aa96-fed7cc57a472" (UID: "db73f5eb-fa9b-4002-aa96-fed7cc57a472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:54.133855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.133827 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca" pod "db73f5eb-fa9b-4002-aa96-fed7cc57a472" (UID: "db73f5eb-fa9b-4002-aa96-fed7cc57a472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:54.133965 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.133863 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b" pod "db73f5eb-fa9b-4002-aa96-fed7cc57a472" (UID: "db73f5eb-fa9b-4002-aa96-fed7cc57a472"). InnerVolumeSpecName "kube-api-access-85v6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:54.233218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.233137 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85v6b\" (UniqueName: \"kubernetes.io/projected/db73f5eb-fa9b-4002-aa96-fed7cc57a472-kube-api-access-85v6b\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:54.233218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.233166 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:54.233218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.233176 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db73f5eb-fa9b-4002-aa96-fed7cc57a472-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:11:54.917864 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.917836 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" Oct 29 23:11:54.917864 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.917855 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn" event={"ID":"db73f5eb-fa9b-4002-aa96-fed7cc57a472","Type":"ContainerDied","Data":"f5b1194c00e6a325852d56f4c88e21d89d7e4f04840bb646fd321ff9011f1377"} Oct 29 23:11:54.918093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:11:54.917896 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b1194c00e6a325852d56f4c88e21d89d7e4f04840bb646fd321ff9011f1377" Oct 29 23:12:04.027142 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:04.027103 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9"] Oct 29 23:12:04.041347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:04.041314 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4472b-9qzj9"] Oct 29 23:12:04.043998 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:04.043969 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd"] Oct 29 23:12:04.046184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:04.046164 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd36dee8-b7jjd"] Oct 29 23:12:05.219951 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:05.219910 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5589ef46-bc4b-4e5f-a94e-369414e4d230" path="/var/lib/kubelet/pods/5589ef46-bc4b-4e5f-a94e-369414e4d230/volumes" Oct 29 23:12:05.220530 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:05.220509 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c181c1f2-c4dc-4687-8f3d-621e0da99b7d" path="/var/lib/kubelet/pods/c181c1f2-c4dc-4687-8f3d-621e0da99b7d/volumes" Oct 29 23:12:08.333379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.333338 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn"] Oct 29 23:12:08.348627 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.348580 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlswn"] Oct 29 23:12:08.375936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.375899 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b"] Oct 29 23:12:08.376309 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.376295 5367 cpu_manager.go:414] "containerMap: removing container" podUID="db73f5eb-fa9b-4002-aa96-fed7cc57a472" containerName="revoke-keys" Oct 29 23:12:08.376352 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.376310 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db73f5eb-fa9b-4002-aa96-fed7cc57a472" containerName="revoke-keys" Oct 29 23:12:08.376405 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.376396 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="db73f5eb-fa9b-4002-aa96-fed7cc57a472" containerName="revoke-keys" Oct 29 23:12:08.379372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.379356 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.396349 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.396271 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:12:08.396651 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.396523 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:12:08.396773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.396722 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:12:08.398591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.398565 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b"] Oct 29 23:12:08.408474 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.408447 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:12:08.470315 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.470278 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.470508 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.470322 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.470508 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.470349 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzxk\" (UniqueName: \"kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.571268 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.571218 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.571268 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.571266 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.571562 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.571293 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzxk\" (UniqueName: \"kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.572051 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.572025 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.573642 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.573618 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.592750 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.592656 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzxk\" (UniqueName: \"kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk\") pod \"maas-api-revoke-sub-3d4156f1-8989b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.692682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.692639 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:08.991557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:08.991519 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b"] Oct 29 23:12:08.993166 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:12:08.993135 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d0d1a8_b89d_4947_8f95_24c48921955b.slice/crio-ab5be7931f9274ca254e9c91d4813810ab43493e72d4766e8838581363385fb0 WatchSource:0}: Error finding container ab5be7931f9274ca254e9c91d4813810ab43493e72d4766e8838581363385fb0: Status 404 returned error can't find the container with id ab5be7931f9274ca254e9c91d4813810ab43493e72d4766e8838581363385fb0 Oct 29 23:12:09.003857 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:09.003820 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" event={"ID":"b0d0d1a8-b89d-4947-8f95-24c48921955b","Type":"ContainerStarted","Data":"ab5be7931f9274ca254e9c91d4813810ab43493e72d4766e8838581363385fb0"} Oct 29 23:12:09.221193 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:09.221158 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db73f5eb-fa9b-4002-aa96-fed7cc57a472" path="/var/lib/kubelet/pods/db73f5eb-fa9b-4002-aa96-fed7cc57a472/volumes" Oct 29 23:12:10.011870 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:10.011785 5367 generic.go:356] "Generic (PLEG): container finished" podID="b0d0d1a8-b89d-4947-8f95-24c48921955b" containerID="1b462758c3f58326ca7410db7861f6f6493807e17a658dd3a83c72396e9ec9f9" exitCode=0 Oct 29 23:12:10.012241 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:10.011875 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" event={"ID":"b0d0d1a8-b89d-4947-8f95-24c48921955b","Type":"ContainerDied","Data":"1b462758c3f58326ca7410db7861f6f6493807e17a658dd3a83c72396e9ec9f9"} Oct 29 23:12:11.031563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.031531 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76"] Oct 29 23:12:11.045155 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.045119 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6"] Oct 29 23:12:11.047859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.047796 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48350f-fxv76"] Oct 29 23:12:11.049819 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.049789 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5271d1b8-mlsv6"] Oct 29 23:12:11.151350 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.151327 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:11.221223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.221176 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37aeebfd-da1f-4dad-b18c-53babfd8bc38" path="/var/lib/kubelet/pods/37aeebfd-da1f-4dad-b18c-53babfd8bc38/volumes" Oct 29 23:12:11.221672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.221653 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f" path="/var/lib/kubelet/pods/b2c99dd9-39f4-4fc9-ad0f-4644f7702b9f/volumes" Oct 29 23:12:11.299296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.299220 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca\") pod \"b0d0d1a8-b89d-4947-8f95-24c48921955b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " Oct 29 23:12:11.299437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.299362 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca\") pod \"b0d0d1a8-b89d-4947-8f95-24c48921955b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " Oct 29 23:12:11.299437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.299393 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk\" (UniqueName: \"kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk\") pod \"b0d0d1a8-b89d-4947-8f95-24c48921955b\" (UID: \"b0d0d1a8-b89d-4947-8f95-24c48921955b\") " Oct 29 23:12:11.299710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.299677 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca" pod "b0d0d1a8-b89d-4947-8f95-24c48921955b" (UID: "b0d0d1a8-b89d-4947-8f95-24c48921955b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:11.301296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.301270 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca" pod "b0d0d1a8-b89d-4947-8f95-24c48921955b" (UID: "b0d0d1a8-b89d-4947-8f95-24c48921955b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:11.301453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.301405 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk" pod "b0d0d1a8-b89d-4947-8f95-24c48921955b" (UID: "b0d0d1a8-b89d-4947-8f95-24c48921955b"). InnerVolumeSpecName "kube-api-access-9nzxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:11.400714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.400658 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.400714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.400692 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d0d1a8-b89d-4947-8f95-24c48921955b-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.400929 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:11.400726 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzxk\" (UniqueName: \"kubernetes.io/projected/b0d0d1a8-b89d-4947-8f95-24c48921955b-kube-api-access-9nzxk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:12:12.026292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:12.026261 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" Oct 29 23:12:12.026494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:12.026294 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b" event={"ID":"b0d0d1a8-b89d-4947-8f95-24c48921955b","Type":"ContainerDied","Data":"ab5be7931f9274ca254e9c91d4813810ab43493e72d4766e8838581363385fb0"} Oct 29 23:12:12.026494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:12.026344 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab5be7931f9274ca254e9c91d4813810ab43493e72d4766e8838581363385fb0" Oct 29 23:12:22.331121 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:12:22.331080 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:12:23.939931 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.939889 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws"] Oct 29 23:12:23.940563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.940540 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b0d0d1a8-b89d-4947-8f95-24c48921955b" containerName="revoke-keys" Oct 29 23:12:23.940613 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.940567 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d0d1a8-b89d-4947-8f95-24c48921955b" containerName="revoke-keys" Oct 29 23:12:23.940759 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.940742 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d0d1a8-b89d-4947-8f95-24c48921955b" containerName="revoke-keys" Oct 29 23:12:23.945762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.945745 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:23.963939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.963914 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7e3223-openshift-default-dockercfg-492qr\"" Oct 29 23:12:23.980183 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:23.980157 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws"] Oct 29 23:12:24.127928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.127893 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.127931 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrvl\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.127960 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.128059 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.128179 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.128258 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.128287 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128435 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.128316 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.128435 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.128335 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229137 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229051 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229137 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229090 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrvl\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229137 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229111 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229137 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229129 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229180 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229226 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229278 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229323 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229506 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229348 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229839 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229532 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229839 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229612 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229966 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229867 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.229966 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.229931 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.230072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.230053 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.231516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.231494 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.231755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.231736 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.249472 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.249441 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.249630 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.249602 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrvl\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl\") pod \"e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.263437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.263408 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:24.582333 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.582308 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws"] Oct 29 23:12:24.583891 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:12:24.583864 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be32365_5d33_4e9f_b291_a81cc513c30f.slice/crio-a56bf71e335f41d52b9627d771bcb1cd5e9982d71ef1930d7aa8e8024a12f8ff WatchSource:0}: Error finding container a56bf71e335f41d52b9627d771bcb1cd5e9982d71ef1930d7aa8e8024a12f8ff: Status 404 returned error can't find the container with id a56bf71e335f41d52b9627d771bcb1cd5e9982d71ef1930d7aa8e8024a12f8ff Oct 29 23:12:24.586002 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.585970 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:12:24.586075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.586037 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:12:24.586075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:24.586066 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:12:25.107371 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:25.107328 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" event={"ID":"2be32365-5d33-4e9f-b291-a81cc513c30f","Type":"ContainerStarted","Data":"7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f"} Oct 29 23:12:25.107824 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:25.107395 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" event={"ID":"2be32365-5d33-4e9f-b291-a81cc513c30f","Type":"ContainerStarted","Data":"a56bf71e335f41d52b9627d771bcb1cd5e9982d71ef1930d7aa8e8024a12f8ff"} Oct 29 23:12:25.154345 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:25.154291 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podStartSLOduration=2.154278716 podStartE2EDuration="2.154278716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:25.152624535 +0000 UTC m=+3412.046699456" watchObservedRunningTime="2026-10-29 23:12:25.154278716 +0000 UTC m=+3412.048353636" Oct 29 23:12:25.266482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:25.266449 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:25.272333 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:25.272307 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:26.113555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:26.113521 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:26.114451 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:26.114432 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:28.875022 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.874983 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd"] Oct 29 23:12:28.879042 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.879018 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:28.910453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.910427 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:12:28.910453 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.910427 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:12:28.910850 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.910818 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7e3223-serving-cert\"" Oct 29 23:12:28.910955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.910856 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 23:12:28.910955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.910904 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 23:12:28.913076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.913057 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd"] Oct 29 23:12:28.975382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.975342 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:28.975551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.975387 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:28.975551 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.975513 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:28.975662 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:28.975555 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwj8n\" (UniqueName: \"kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.076085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.076048 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.076085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.076088 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.076335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.076130 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.076335 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.076289 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwj8n\" (UniqueName: \"kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.076986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.076964 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.078585 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.078567 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.078712 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.078666 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.097302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.097278 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwj8n\" (UniqueName: \"kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n\") pod \"maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.193436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.193348 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:29.511561 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.511473 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd"] Oct 29 23:12:29.528873 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:12:29.528831 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309b0dd1_db89_4f14_849a_cba10efbe2a8.slice/crio-b6bd90555550c5a79d2da62bac85a9fbc1635715aa9b861859a647d295383356 WatchSource:0}: Error finding container b6bd90555550c5a79d2da62bac85a9fbc1635715aa9b861859a647d295383356: Status 404 returned error can't find the container with id b6bd90555550c5a79d2da62bac85a9fbc1635715aa9b861859a647d295383356 Oct 29 23:12:29.530430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:29.530405 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:12:30.139052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:30.138955 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" event={"ID":"309b0dd1-db89-4f14-849a-cba10efbe2a8","Type":"ContainerStarted","Data":"1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1"} Oct 29 23:12:30.139052 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:30.139006 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" event={"ID":"309b0dd1-db89-4f14-849a-cba10efbe2a8","Type":"ContainerStarted","Data":"b6bd90555550c5a79d2da62bac85a9fbc1635715aa9b861859a647d295383356"} Oct 29 23:12:30.139498 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:30.139060 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:30.185585 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:30.185533 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" podStartSLOduration=1.854073213 podStartE2EDuration="2.185519198s" totalImagesPullingTime="331.445985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:30.184449538 +0000 UTC m=+3417.078524470" watchObservedRunningTime="2026-10-29 23:12:30.185519198 +0000 UTC m=+3417.079594119" Oct 29 23:12:36.151224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:36.151191 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:12:40.106394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.106356 5367 scope.go:122] "RemoveContainer" containerID="db85d9e9d0b3e22ee1458cfd00d5502b552b0e04e5dc9510db837f35c3f6e6b3" Oct 29 23:12:40.118112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.118087 5367 scope.go:122] "RemoveContainer" containerID="94d984c3b0ec39ee5a89f25b2e63fe9e8f7885b23ada48b33b1e54424c08dff6" Oct 29 23:12:40.127129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.127103 5367 scope.go:122] "RemoveContainer" containerID="6e87ddb79e7f8d9750052a70355a99489c44f4fef0c0b86a1d6ccd18695bfbf0" Oct 29 23:12:40.138246 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.138221 5367 scope.go:122] "RemoveContainer" containerID="4b57a4a2727146cd6fa9fa949b03067fbe7b0806972080627042cd72fbfa4f92" Oct 29 23:12:40.146986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.146963 5367 scope.go:122] "RemoveContainer" containerID="504f22e9e017e9a96f422599a761106e655d621c27b91aa39650254520e24066" Oct 29 23:12:40.155596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.155568 5367 scope.go:122] "RemoveContainer" containerID="7fb755eb5fbc244e5068de33584f892c2c240163344ce7b44e9db2022305d65e" Oct 29 23:12:40.163658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.163634 5367 scope.go:122] "RemoveContainer" containerID="89395d969042304c185b37cec66453788b74ff489af6acd8d97b0e0c857e0cad" Oct 29 23:12:40.173416 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.173398 5367 scope.go:122] "RemoveContainer" containerID="07a02ecc5e7359a972c9be5c9ab0961f9f13fb7306330ae9451403813d7fa773" Oct 29 23:12:40.755843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.755810 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq"] Oct 29 23:12:40.761322 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.761302 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.778437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.778412 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7e3223\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.792009 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.791987 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7e3223\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.792139 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.792080 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7e3223\"/\"simulator-single-main-7e3223-kserve-self-signed-certs\"" Oct 29 23:12:40.792199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.792165 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7e3223\"/\"default-dockercfg-glxcw\"" Oct 29 23:12:40.795067 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.795049 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq"] Oct 29 23:12:40.795681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.795658 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.795822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.795801 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.795890 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.795855 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.795994 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.795978 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.796031 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.796007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbrq\" (UniqueName: \"kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.796031 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.796027 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.896747 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.896685 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.896941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.896770 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbrq\" (UniqueName: \"kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.896941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.896800 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.896941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.896886 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.896941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.896935 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.897198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.896986 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.897198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.897171 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.897318 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.897295 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.897415 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.897392 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.899220 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.899194 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.899494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.899478 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:40.917773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:40.917742 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbrq\" (UniqueName: \"kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq\") pod \"simulator-single-main-7e3223-kserve-646d5d4764-nptlq\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:41.074345 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:41.074308 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:41.378741 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:41.378684 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq"] Oct 29 23:12:41.379915 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:12:41.379888 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20786ccc_7195_43c3_b03f_d2f7b30ce978.slice/crio-06d75505cf95ddffcf7711405ef0dcb73a50a528616454b83b71fb51ce6e3506 WatchSource:0}: Error finding container 06d75505cf95ddffcf7711405ef0dcb73a50a528616454b83b71fb51ce6e3506: Status 404 returned error can't find the container with id 06d75505cf95ddffcf7711405ef0dcb73a50a528616454b83b71fb51ce6e3506 Oct 29 23:12:41.687291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:41.686895 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": read tcp 10.133.0.2:57456->10.133.1.149:15021: read: connection reset by peer" start-of-body= Oct 29 23:12:41.687291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:41.686968 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": read tcp 10.133.0.2:57456->10.133.1.149:15021: read: connection reset by peer" Oct 29 23:12:42.236367 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.236335 5367 generic.go:356] "Generic (PLEG): container finished" podID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerID="7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f" exitCode=0 Oct 29 23:12:42.236564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.236418 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" event={"ID":"2be32365-5d33-4e9f-b291-a81cc513c30f","Type":"ContainerDied","Data":"7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f"} Oct 29 23:12:42.236897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.236875 5367 scope.go:122] "RemoveContainer" containerID="7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f" Oct 29 23:12:42.238072 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.238038 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:12:42.238205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.238115 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:12:42.238205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.238131 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" event={"ID":"20786ccc-7195-43c3-b03f-d2f7b30ce978","Type":"ContainerStarted","Data":"15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0"} Oct 29 23:12:42.238205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.238156 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" event={"ID":"20786ccc-7195-43c3-b03f-d2f7b30ce978","Type":"ContainerStarted","Data":"06d75505cf95ddffcf7711405ef0dcb73a50a528616454b83b71fb51ce6e3506"} Oct 29 23:12:42.238205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.238160 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:12:42.342834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:42.342773 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" podStartSLOduration=2.34275667 podStartE2EDuration="2.34275667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:42.341513636 +0000 UTC m=+3429.235588559" watchObservedRunningTime="2026-10-29 23:12:42.34275667 +0000 UTC m=+3429.236831591" Oct 29 23:12:43.077852 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:43.077809 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:43.247206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:43.247169 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" event={"ID":"2be32365-5d33-4e9f-b291-a81cc513c30f","Type":"ContainerStarted","Data":"eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1"} Oct 29 23:12:43.266163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:43.266125 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:43.266438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:43.266403 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body= Oct 29 23:12:43.266676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:43.266650 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" Oct 29 23:12:44.267200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:44.267107 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body= Oct 29 23:12:44.267820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:44.267788 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" Oct 29 23:12:45.277724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:45.277260 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body= Oct 29 23:12:45.277724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:45.277340 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" Oct 29 23:12:46.274723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:46.272049 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body= Oct 29 23:12:46.274723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:46.272124 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" Oct 29 23:12:47.274722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:47.269051 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body= Oct 29 23:12:47.274722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:47.269111 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" Oct 29 23:12:48.263877 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:48.263835 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body= Oct 29 23:12:48.264131 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:48.263911 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" Oct 29 23:12:49.268036 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:49.267998 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:49.268471 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:49.268252 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:49.268985 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:49.268965 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:12:51.075152 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:51.075106 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:51.087940 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:51.087908 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:51.330114 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:51.330010 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:12:52.332627 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:12:52.332591 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:12:58.028589 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:58.028540 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q"] Oct 29 23:12:58.042485 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:58.042453 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6cn9q"] Oct 29 23:12:59.220194 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:12:59.220157 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1827a175-3964-4836-8607-420e43f04878" path="/var/lib/kubelet/pods/1827a175-3964-4836-8607-420e43f04878/volumes" Oct 29 23:13:19.018163 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:19.018131 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq"] Oct 29 23:13:22.330218 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:13:22.330182 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:13:24.817393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:24.817355 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd"] Oct 29 23:13:24.817909 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:24.817573 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" podUID="309b0dd1-db89-4f14-849a-cba10efbe2a8" containerName="maas-api" containerID="cri-o://1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1" gracePeriod=30 Oct 29 23:13:25.284246 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.284215 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:13:25.367766 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.367668 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls\") pod \"309b0dd1-db89-4f14-849a-cba10efbe2a8\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " Oct 29 23:13:25.367766 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.367749 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n\" (UniqueName: \"kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n\") pod \"309b0dd1-db89-4f14-849a-cba10efbe2a8\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " Oct 29 23:13:25.367966 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.367871 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca\") pod \"309b0dd1-db89-4f14-849a-cba10efbe2a8\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " Oct 29 23:13:25.368003 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.367977 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs\") pod \"309b0dd1-db89-4f14-849a-cba10efbe2a8\" (UID: \"309b0dd1-db89-4f14-849a-cba10efbe2a8\") " Oct 29 23:13:25.368271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.368228 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca" pod "309b0dd1-db89-4f14-849a-cba10efbe2a8" (UID: "309b0dd1-db89-4f14-849a-cba10efbe2a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:25.369851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.369821 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls" pod "309b0dd1-db89-4f14-849a-cba10efbe2a8" (UID: "309b0dd1-db89-4f14-849a-cba10efbe2a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:25.369996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.369960 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs" pod "309b0dd1-db89-4f14-849a-cba10efbe2a8" (UID: "309b0dd1-db89-4f14-849a-cba10efbe2a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:25.370055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.370031 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n" pod "309b0dd1-db89-4f14-849a-cba10efbe2a8" (UID: "309b0dd1-db89-4f14-849a-cba10efbe2a8"). InnerVolumeSpecName "kube-api-access-hwj8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:25.468887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.468848 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b0dd1-db89-4f14-849a-cba10efbe2a8-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:25.468887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.468883 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:25.468887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.468893 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309b0dd1-db89-4f14-849a-cba10efbe2a8-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:25.469151 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.468901 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwj8n\" (UniqueName: \"kubernetes.io/projected/309b0dd1-db89-4f14-849a-cba10efbe2a8-kube-api-access-hwj8n\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:25.547499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.547460 5367 generic.go:356] "Generic (PLEG): container finished" podID="309b0dd1-db89-4f14-849a-cba10efbe2a8" containerID="1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1" exitCode=0 Oct 29 23:13:25.547676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.547505 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" Oct 29 23:13:25.547676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.547535 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" event={"ID":"309b0dd1-db89-4f14-849a-cba10efbe2a8","Type":"ContainerDied","Data":"1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1"} Oct 29 23:13:25.547676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.547590 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd" event={"ID":"309b0dd1-db89-4f14-849a-cba10efbe2a8","Type":"ContainerDied","Data":"b6bd90555550c5a79d2da62bac85a9fbc1635715aa9b861859a647d295383356"} Oct 29 23:13:25.547676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.547611 5367 scope.go:122] "RemoveContainer" containerID="1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1" Oct 29 23:13:25.558459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.558437 5367 scope.go:122] "RemoveContainer" containerID="1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1" Oct 29 23:13:25.558745 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:13:25.558725 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1\": container with ID starting with 1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1 not found: ID does not exist" containerID="1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1" Oct 29 23:13:25.558801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.558758 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1"} err="failed to get container status \"1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1\": rpc error: code = NotFound desc = could not find container \"1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1\": container with ID starting with 1fad7e9c8d5c6adbc660c62480c4ae3aa3cda66448f660864f78e980798341b1 not found: ID does not exist" Oct 29 23:13:25.605898 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.605866 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd"] Oct 29 23:13:25.618834 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:25.618758 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e3223-54d658bdc5-7bcrd"] Oct 29 23:13:27.220465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:27.220433 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309b0dd1-db89-4f14-849a-cba10efbe2a8" path="/var/lib/kubelet/pods/309b0dd1-db89-4f14-849a-cba10efbe2a8/volumes" Oct 29 23:13:32.776188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:32.776149 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws"] Oct 29 23:13:32.776883 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:32.776397 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" containerID="cri-o://eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1" gracePeriod=30 Oct 29 23:13:34.038207 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.038136 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" podUID="20786ccc-7195-43c3-b03f-d2f7b30ce978" containerName="main" containerID="cri-o://15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0" gracePeriod=45 Oct 29 23:13:34.057463 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.057431 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b"] Oct 29 23:13:34.071188 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.071156 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8989b"] Oct 29 23:13:34.100814 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.100779 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6"] Oct 29 23:13:34.101214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.101201 5367 cpu_manager.go:414] "containerMap: removing container" podUID="309b0dd1-db89-4f14-849a-cba10efbe2a8" containerName="maas-api" Oct 29 23:13:34.101261 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.101216 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309b0dd1-db89-4f14-849a-cba10efbe2a8" containerName="maas-api" Oct 29 23:13:34.101302 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.101294 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="309b0dd1-db89-4f14-849a-cba10efbe2a8" containerName="maas-api" Oct 29 23:13:34.104520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.104501 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.122440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.122388 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:13:34.122440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.122388 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:13:34.123443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.122819 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:13:34.123443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.122985 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:13:34.126446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.126421 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6"] Oct 29 23:13:34.152668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.152636 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.152864 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.152725 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cptt\" (UniqueName: \"kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.152936 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.152882 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.253510 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.253480 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.253722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.253560 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.253853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.253825 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cptt\" (UniqueName: \"kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.254235 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.254213 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.255968 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.255945 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.270495 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.270462 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:13:34.270603 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.270512 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:13:34.274663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.274643 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cptt\" (UniqueName: \"kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt\") pod \"maas-api-revoke-sub-3d4156f1-ncfl6\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.289897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.289844 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:13:34.355206 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355175 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir\") pod \"20786ccc-7195-43c3-b03f-d2f7b30ce978\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " Oct 29 23:13:34.355404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355243 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm\") pod \"20786ccc-7195-43c3-b03f-d2f7b30ce978\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " Oct 29 23:13:34.355404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355309 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home\") pod \"20786ccc-7195-43c3-b03f-d2f7b30ce978\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " Oct 29 23:13:34.355404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355331 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq\" (UniqueName: \"kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq\") pod \"20786ccc-7195-43c3-b03f-d2f7b30ce978\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " Oct 29 23:13:34.355404 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355360 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs\" (UniqueName: \"kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs\") pod \"20786ccc-7195-43c3-b03f-d2f7b30ce978\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " Oct 29 23:13:34.356113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355407 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache\") pod \"20786ccc-7195-43c3-b03f-d2f7b30ce978\" (UID: \"20786ccc-7195-43c3-b03f-d2f7b30ce978\") " Oct 29 23:13:34.356113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355480 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir" pod "20786ccc-7195-43c3-b03f-d2f7b30ce978" (UID: "20786ccc-7195-43c3-b03f-d2f7b30ce978"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:34.356113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355579 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home" pod "20786ccc-7195-43c3-b03f-d2f7b30ce978" (UID: "20786ccc-7195-43c3-b03f-d2f7b30ce978"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:34.356113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355777 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:34.356113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355799 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:34.356113 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.355856 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache" pod "20786ccc-7195-43c3-b03f-d2f7b30ce978" (UID: "20786ccc-7195-43c3-b03f-d2f7b30ce978"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:34.357736 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.357681 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs" pod "20786ccc-7195-43c3-b03f-d2f7b30ce978" (UID: "20786ccc-7195-43c3-b03f-d2f7b30ce978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:34.357856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.357802 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm" pod "20786ccc-7195-43c3-b03f-d2f7b30ce978" (UID: "20786ccc-7195-43c3-b03f-d2f7b30ce978"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:34.357856 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.357802 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq" pod "20786ccc-7195-43c3-b03f-d2f7b30ce978" (UID: "20786ccc-7195-43c3-b03f-d2f7b30ce978"). InnerVolumeSpecName "kube-api-access-jcbrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:34.436673 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.436642 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:34.456435 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.456405 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcbrq\" (UniqueName: \"kubernetes.io/projected/20786ccc-7195-43c3-b03f-d2f7b30ce978-kube-api-access-jcbrq\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:34.456555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.456437 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20786ccc-7195-43c3-b03f-d2f7b30ce978-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:34.456555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.456454 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:34.456555 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.456465 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20786ccc-7195-43c3-b03f-d2f7b30ce978-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:34.602990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.602956 5367 generic.go:356] "Generic (PLEG): container finished" podID="20786ccc-7195-43c3-b03f-d2f7b30ce978" containerID="15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0" exitCode=0 Oct 29 23:13:34.603185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.603003 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" Oct 29 23:13:34.603185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.603042 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" event={"ID":"20786ccc-7195-43c3-b03f-d2f7b30ce978","Type":"ContainerDied","Data":"15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0"} Oct 29 23:13:34.603185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.603087 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq" event={"ID":"20786ccc-7195-43c3-b03f-d2f7b30ce978","Type":"ContainerDied","Data":"06d75505cf95ddffcf7711405ef0dcb73a50a528616454b83b71fb51ce6e3506"} Oct 29 23:13:34.603185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.603104 5367 scope.go:122] "RemoveContainer" containerID="15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0" Oct 29 23:13:34.613337 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.613316 5367 scope.go:122] "RemoveContainer" containerID="15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0" Oct 29 23:13:34.613624 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:13:34.613591 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0\": container with ID starting with 15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0 not found: ID does not exist" containerID="15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0" Oct 29 23:13:34.613714 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.613622 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0"} err="failed to get container status \"15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0\": rpc error: code = NotFound desc = could not find container \"15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0\": container with ID starting with 15dd86a7397c598355c9af040733722f4791d209b5ad99d61413aedad0dd7fd0 not found: ID does not exist" Oct 29 23:13:34.659085 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.659054 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq"] Oct 29 23:13:34.663184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.663159 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7e3223/simulator-single-main-7e3223-kserve-646d5d4764-nptlq"] Oct 29 23:13:34.736172 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:34.736148 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6"] Oct 29 23:13:34.737596 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:13:34.737568 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755a9360_3650_4b4a_8317_6f12133b664a.slice/crio-4876709e39283b23b373b26bc3da0bb0ab7aa1a160e31565c04429d982cab7b2 WatchSource:0}: Error finding container 4876709e39283b23b373b26bc3da0bb0ab7aa1a160e31565c04429d982cab7b2: Status 404 returned error can't find the container with id 4876709e39283b23b373b26bc3da0bb0ab7aa1a160e31565c04429d982cab7b2 Oct 29 23:13:35.228054 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:35.227962 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20786ccc-7195-43c3-b03f-d2f7b30ce978" path="/var/lib/kubelet/pods/20786ccc-7195-43c3-b03f-d2f7b30ce978/volumes" Oct 29 23:13:35.228421 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:35.228397 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d0d1a8-b89d-4947-8f95-24c48921955b" path="/var/lib/kubelet/pods/b0d0d1a8-b89d-4947-8f95-24c48921955b/volumes" Oct 29 23:13:35.610539 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:35.610506 5367 generic.go:356] "Generic (PLEG): container finished" podID="755a9360-3650-4b4a-8317-6f12133b664a" containerID="f292440bc47cb406749993b25719cd92309d9371edf3372ec2951ad30414a904" exitCode=0 Oct 29 23:13:35.610721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:35.610576 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" event={"ID":"755a9360-3650-4b4a-8317-6f12133b664a","Type":"ContainerDied","Data":"f292440bc47cb406749993b25719cd92309d9371edf3372ec2951ad30414a904"} Oct 29 23:13:35.610721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:35.610628 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" event={"ID":"755a9360-3650-4b4a-8317-6f12133b664a","Type":"ContainerStarted","Data":"4876709e39283b23b373b26bc3da0bb0ab7aa1a160e31565c04429d982cab7b2"} Oct 29 23:13:36.748370 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.748341 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:36.883004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.882910 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca\") pod \"755a9360-3650-4b4a-8317-6f12133b664a\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " Oct 29 23:13:36.883191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.883003 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca\") pod \"755a9360-3650-4b4a-8317-6f12133b664a\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " Oct 29 23:13:36.883191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.883043 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt\" (UniqueName: \"kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt\") pod \"755a9360-3650-4b4a-8317-6f12133b664a\" (UID: \"755a9360-3650-4b4a-8317-6f12133b664a\") " Oct 29 23:13:36.883352 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.883325 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca" pod "755a9360-3650-4b4a-8317-6f12133b664a" (UID: "755a9360-3650-4b4a-8317-6f12133b664a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:36.885099 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.885074 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca" pod "755a9360-3650-4b4a-8317-6f12133b664a" (UID: "755a9360-3650-4b4a-8317-6f12133b664a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:36.885199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.885139 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt" pod "755a9360-3650-4b4a-8317-6f12133b664a" (UID: "755a9360-3650-4b4a-8317-6f12133b664a"). InnerVolumeSpecName "kube-api-access-4cptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:36.984443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.984410 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:36.984443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.984437 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755a9360-3650-4b4a-8317-6f12133b664a-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:36.984443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:36.984447 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cptt\" (UniqueName: \"kubernetes.io/projected/755a9360-3650-4b4a-8317-6f12133b664a-kube-api-access-4cptt\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:37.624802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:37.624770 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" Oct 29 23:13:37.624802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:37.624771 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6" event={"ID":"755a9360-3650-4b4a-8317-6f12133b664a","Type":"ContainerDied","Data":"4876709e39283b23b373b26bc3da0bb0ab7aa1a160e31565c04429d982cab7b2"} Oct 29 23:13:37.625025 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:37.624846 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4876709e39283b23b373b26bc3da0bb0ab7aa1a160e31565c04429d982cab7b2" Oct 29 23:13:38.028611 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.028583 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:13:38.094976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.094945 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.094997 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095182 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095080 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095209 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095256 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095294 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095321 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095406 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095452 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy\") pod \"2be32365-5d33-4e9f-b291-a81cc513c30f\" (UID: \"2be32365-5d33-4e9f-b291-a81cc513c30f\") " Oct 29 23:13:38.095564 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095437 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:38.095778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095608 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:38.095778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095661 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:38.095778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095677 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:38.095938 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095893 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.095938 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095915 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.095938 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095929 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.096056 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.095943 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.096277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.096251 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:38.097667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.097638 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:38.097894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.097870 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "kube-api-access-lvrvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:38.097961 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.097899 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:13:38.098921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.098897 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy" pod "2be32365-5d33-4e9f-b291-a81cc513c30f" (UID: "2be32365-5d33-4e9f-b291-a81cc513c30f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:38.196493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.196407 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrvl\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-kube-api-access-lvrvl\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.196493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.196440 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.196493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.196450 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.196493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.196458 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be32365-5d33-4e9f-b291-a81cc513c30f-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.196493 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.196466 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be32365-5d33-4e9f-b291-a81cc513c30f-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:13:38.633083 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.633044 5367 generic.go:356] "Generic (PLEG): container finished" podID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerID="eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1" exitCode=0 Oct 29 23:13:38.633382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.633092 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" Oct 29 23:13:38.633382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.633122 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" event={"ID":"2be32365-5d33-4e9f-b291-a81cc513c30f","Type":"ContainerDied","Data":"eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1"} Oct 29 23:13:38.633382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.633175 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws" event={"ID":"2be32365-5d33-4e9f-b291-a81cc513c30f","Type":"ContainerDied","Data":"a56bf71e335f41d52b9627d771bcb1cd5e9982d71ef1930d7aa8e8024a12f8ff"} Oct 29 23:13:38.633382 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.633196 5367 scope.go:122] "RemoveContainer" containerID="eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1" Oct 29 23:13:38.643357 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.643336 5367 scope.go:122] "RemoveContainer" containerID="7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f" Oct 29 23:13:38.652480 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.652461 5367 scope.go:122] "RemoveContainer" containerID="eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1" Oct 29 23:13:38.652761 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:13:38.652745 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1\": container with ID starting with eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1 not found: ID does not exist" containerID="eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1" Oct 29 23:13:38.652817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.652771 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1"} err="failed to get container status \"eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1\": rpc error: code = NotFound desc = could not find container \"eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1\": container with ID starting with eb4a47cf069dd250225e4b69398d3dc553248e222bc7ba3b1983cb2c7db9a0b1 not found: ID does not exist" Oct 29 23:13:38.652817 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.652789 5367 scope.go:122] "RemoveContainer" containerID="7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f" Oct 29 23:13:38.653020 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:13:38.652999 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f\": container with ID starting with 7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f not found: ID does not exist" containerID="7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f" Oct 29 23:13:38.653067 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.653028 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f"} err="failed to get container status \"7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f\": rpc error: code = NotFound desc = could not find container \"7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f\": container with ID starting with 7ba77f6e7177f47de91d9279cda74be831e02a85d27cdc19509218ca8e3e712f not found: ID does not exist" Oct 29 23:13:38.703941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.703907 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws"] Oct 29 23:13:38.717499 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:38.717468 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e3223-openshift-default-7788864c8fcv8ws"] Oct 29 23:13:39.220544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:39.220511 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" path="/var/lib/kubelet/pods/2be32365-5d33-4e9f-b291-a81cc513c30f/volumes" Oct 29 23:13:40.231791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:40.231763 5367 scope.go:122] "RemoveContainer" containerID="fcfee064f9c9179972185575e9b08f838860d7c0796d15557262f0bee2606b9f" Oct 29 23:13:40.240662 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:40.240640 5367 scope.go:122] "RemoveContainer" containerID="07891d5ef54b00821eec34641766c736cb539228ac5186b7fffa87b9f224bdad" Oct 29 23:13:40.248769 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:40.248746 5367 scope.go:122] "RemoveContainer" containerID="1fefc68eb64543eced577322d94d3148092f29b0e5e603936983af420bcf1994" Oct 29 23:13:40.256691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:40.256664 5367 scope.go:122] "RemoveContainer" containerID="e6d38e70c2e1a0a28b28ab6f508030e2341931be2b1c55c9a705ca06612b6525" Oct 29 23:13:40.267069 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:40.267050 5367 scope.go:122] "RemoveContainer" containerID="89ed0fa737b8e2231c50776fbaa0883e1aefe80aa0309dc4b6a33e6c153866e3" Oct 29 23:13:40.275223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:40.275205 5367 scope.go:122] "RemoveContainer" containerID="9db7080255e1ae18fd9b249547b6d38687af8ad2fd4a5d93404d21037e168bd9" Oct 29 23:13:50.027868 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:50.027823 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z"] Oct 29 23:13:50.041632 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:50.041593 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mvh5z"] Oct 29 23:13:51.219458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:13:51.219422 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4127ef-9331-467e-855f-cd28e5f2db37" path="/var/lib/kubelet/pods/0a4127ef-9331-467e-855f-cd28e5f2db37/volumes" Oct 29 23:13:52.333508 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:13:52.333461 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:14:11.030071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:11.030034 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p"] Oct 29 23:14:11.043112 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:11.043081 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-clw7p"] Oct 29 23:14:11.219567 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:11.219530 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0e7199-f59f-4036-b778-3fa6facdca73" path="/var/lib/kubelet/pods/5b0e7199-f59f-4036-b778-3fa6facdca73/volumes" Oct 29 23:14:22.329955 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:14:22.329913 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:14:23.032713 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:23.032664 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn"] Oct 29 23:14:23.046438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:23.046397 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7hfvn"] Oct 29 23:14:23.220750 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:23.220718 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63cc360-1b42-4226-9df0-fe1845ce30f7" path="/var/lib/kubelet/pods/c63cc360-1b42-4226-9df0-fe1845ce30f7/volumes" Oct 29 23:14:40.326070 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:40.326035 5367 scope.go:122] "RemoveContainer" containerID="1a54c2b7c7473836d50078bb2bf7bf4b17df94d9593512ba52d676ac141bc641" Oct 29 23:14:51.051663 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:51.051617 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl"] Oct 29 23:14:51.065146 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:51.065120 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzgtl"] Oct 29 23:14:51.219595 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:14:51.219560 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab7be4f-bf30-4b5a-9d93-90d952b53565" path="/var/lib/kubelet/pods/bab7be4f-bf30-4b5a-9d93-90d952b53565/volumes" Oct 29 23:14:52.330340 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:14:52.330296 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:15:00.145074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145040 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7"] Oct 29 23:15:00.145465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145394 5367 cpu_manager.go:414] "containerMap: removing container" podUID="755a9360-3650-4b4a-8317-6f12133b664a" containerName="revoke-keys" Oct 29 23:15:00.145465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145405 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755a9360-3650-4b4a-8317-6f12133b664a" containerName="revoke-keys" Oct 29 23:15:00.145465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145416 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" Oct 29 23:15:00.145465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145421 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" Oct 29 23:15:00.145465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145433 5367 cpu_manager.go:414] "containerMap: removing container" podUID="20786ccc-7195-43c3-b03f-d2f7b30ce978" containerName="main" Oct 29 23:15:00.145465 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145437 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20786ccc-7195-43c3-b03f-d2f7b30ce978" containerName="main" Oct 29 23:15:00.145688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145507 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="755a9360-3650-4b4a-8317-6f12133b664a" containerName="revoke-keys" Oct 29 23:15:00.145688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145516 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" Oct 29 23:15:00.145688 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.145523 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="20786ccc-7195-43c3-b03f-d2f7b30ce978" containerName="main" Oct 29 23:15:00.149942 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.149922 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.180334 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.180306 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:15:00.180511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.180336 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:15:00.180511 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.180451 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:15:00.182542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.182520 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7"] Oct 29 23:15:00.270071 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.270015 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpp95\" (UniqueName: \"kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95\") pod \"maas-api-key-cleanup-29888595-5mzz7\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.270341 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.270312 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-5mzz7\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.371215 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.371170 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-5mzz7\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.371393 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.371229 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpp95\" (UniqueName: \"kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95\") pod \"maas-api-key-cleanup-29888595-5mzz7\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.371853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.371834 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-5mzz7\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.392582 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.392550 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpp95\" (UniqueName: \"kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95\") pod \"maas-api-key-cleanup-29888595-5mzz7\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.462329 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.462249 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:00.759016 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:00.758994 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7"] Oct 29 23:15:00.760733 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:15:00.760685 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42d55d7_f6ca_4b76_a5e4_e0d954849cbd.slice/crio-7efb1c38fb1b29dbaa1487a8316b606e723c5cc95d3496dcacdd8398e43c935f WatchSource:0}: Error finding container 7efb1c38fb1b29dbaa1487a8316b606e723c5cc95d3496dcacdd8398e43c935f: Status 404 returned error can't find the container with id 7efb1c38fb1b29dbaa1487a8316b606e723c5cc95d3496dcacdd8398e43c935f Oct 29 23:15:01.029745 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:01.029651 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q"] Oct 29 23:15:01.036685 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:01.036655 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7789q"] Oct 29 23:15:01.074668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:01.074637 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" event={"ID":"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd","Type":"ContainerStarted","Data":"a74ab6305953c2e30fbf0c83bf7b51ae17c42a73fb212a0e5da881a983cf914b"} Oct 29 23:15:01.074842 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:01.074687 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" event={"ID":"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd","Type":"ContainerStarted","Data":"7efb1c38fb1b29dbaa1487a8316b606e723c5cc95d3496dcacdd8398e43c935f"} Oct 29 23:15:01.117841 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:01.117796 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" podStartSLOduration=1.117783361 podStartE2EDuration="1.117783361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.116311833 +0000 UTC m=+3568.010386755" watchObservedRunningTime="2026-10-29 23:15:01.117783361 +0000 UTC m=+3568.011858280" Oct 29 23:15:01.221450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:01.221419 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494428d5-70e2-4013-8f56-29d416d542f1" path="/var/lib/kubelet/pods/494428d5-70e2-4013-8f56-29d416d542f1/volumes" Oct 29 23:15:02.085028 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:02.084985 5367 generic.go:356] "Generic (PLEG): container finished" podID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerID="a74ab6305953c2e30fbf0c83bf7b51ae17c42a73fb212a0e5da881a983cf914b" exitCode=60 Oct 29 23:15:02.085249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:02.085101 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" event={"ID":"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd","Type":"ContainerDied","Data":"a74ab6305953c2e30fbf0c83bf7b51ae17c42a73fb212a0e5da881a983cf914b"} Oct 29 23:15:02.085600 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:02.085578 5367 scope.go:122] "RemoveContainer" containerID="a74ab6305953c2e30fbf0c83bf7b51ae17c42a73fb212a0e5da881a983cf914b" Oct 29 23:15:03.093226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:03.093192 5367 generic.go:356] "Generic (PLEG): container finished" podID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerID="612b1a41c7a2f0ff3b2da6702c868071b87379a5fa5e5e809affe8e377146568" exitCode=60 Oct 29 23:15:03.093717 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:03.093275 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" event={"ID":"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd","Type":"ContainerDied","Data":"612b1a41c7a2f0ff3b2da6702c868071b87379a5fa5e5e809affe8e377146568"} Oct 29 23:15:03.093717 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:03.093349 5367 scope.go:122] "RemoveContainer" containerID="a74ab6305953c2e30fbf0c83bf7b51ae17c42a73fb212a0e5da881a983cf914b" Oct 29 23:15:03.093717 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:03.093567 5367 scope.go:122] "RemoveContainer" containerID="612b1a41c7a2f0ff3b2da6702c868071b87379a5fa5e5e809affe8e377146568" Oct 29 23:15:03.093875 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:15:03.093854 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-5mzz7_odh-ai-gateway-infra(f42d55d7-f6ca-4b76-a5e4-e0d954849cbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" Oct 29 23:15:04.100497 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:04.100453 5367 scope.go:122] "RemoveContainer" containerID="612b1a41c7a2f0ff3b2da6702c868071b87379a5fa5e5e809affe8e377146568" Oct 29 23:15:04.100947 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:15:04.100689 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-5mzz7_odh-ai-gateway-infra(f42d55d7-f6ca-4b76-a5e4-e0d954849cbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" Oct 29 23:15:14.214253 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:14.214224 5367 scope.go:122] "RemoveContainer" containerID="612b1a41c7a2f0ff3b2da6702c868071b87379a5fa5e5e809affe8e377146568" Oct 29 23:15:15.160400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:15.160363 5367 generic.go:356] "Generic (PLEG): container finished" podID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerID="7bb1ee83704ea044de1ec47bcd0a60b9d1890af84cbd0cb0d295daf489c7f61b" exitCode=60 Oct 29 23:15:15.160605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:15.160449 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" event={"ID":"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd","Type":"ContainerDied","Data":"7bb1ee83704ea044de1ec47bcd0a60b9d1890af84cbd0cb0d295daf489c7f61b"} Oct 29 23:15:15.160605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:15.160532 5367 scope.go:122] "RemoveContainer" containerID="612b1a41c7a2f0ff3b2da6702c868071b87379a5fa5e5e809affe8e377146568" Oct 29 23:15:15.160887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:15.160870 5367 scope.go:122] "RemoveContainer" containerID="7bb1ee83704ea044de1ec47bcd0a60b9d1890af84cbd0cb0d295daf489c7f61b" Oct 29 23:15:15.161155 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:15:15.161127 5367 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-5mzz7_odh-ai-gateway-infra(f42d55d7-f6ca-4b76-a5e4-e0d954849cbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" Oct 29 23:15:16.215347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.215311 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7"] Oct 29 23:15:16.639262 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.639227 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:16.742120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.742091 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95\" (UniqueName: \"kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95\") pod \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " Oct 29 23:15:16.742295 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.742132 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca\") pod \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\" (UID: \"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd\") " Oct 29 23:15:16.742751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.742711 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca" pod "f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" (UID: "f42d55d7-f6ca-4b76-a5e4-e0d954849cbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:16.744458 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.744431 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95" pod "f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" (UID: "f42d55d7-f6ca-4b76-a5e4-e0d954849cbd"). InnerVolumeSpecName "kube-api-access-fpp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:16.843520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.843493 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpp95\" (UniqueName: \"kubernetes.io/projected/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-kube-api-access-fpp95\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:16.843751 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:16.843737 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd-maas-api-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:17.225456 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:17.225237 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" Oct 29 23:15:17.229466 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:17.229430 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7" event={"ID":"f42d55d7-f6ca-4b76-a5e4-e0d954849cbd","Type":"ContainerDied","Data":"7efb1c38fb1b29dbaa1487a8316b606e723c5cc95d3496dcacdd8398e43c935f"} Oct 29 23:15:17.229608 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:17.229515 5367 scope.go:122] "RemoveContainer" containerID="7bb1ee83704ea044de1ec47bcd0a60b9d1890af84cbd0cb0d295daf489c7f61b" Oct 29 23:15:17.285014 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:17.284984 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7"] Oct 29 23:15:17.293403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:17.293379 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-5mzz7"] Oct 29 23:15:18.930549 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930508 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg"] Oct 29 23:15:18.930976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930878 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:18.930976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930891 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:18.930976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930905 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:18.930976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930910 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:18.930976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930936 5367 cpu_manager.go:414] "containerMap: removing container" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" Oct 29 23:15:18.930976 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.930942 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" Oct 29 23:15:18.931175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.931001 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:18.931175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.931014 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be32365-5d33-4e9f-b291-a81cc513c30f" containerName="istio-proxy" Oct 29 23:15:18.931175 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.931023 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:18.935326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.935308 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:18.952548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.952528 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:15:18.952693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.952643 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:15:18.961658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.961623 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:15:18.961658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.961627 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:15:18.964535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:18.964504 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg"] Oct 29 23:15:19.065749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.065693 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.065895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.065814 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.065895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.065867 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv746\" (UniqueName: \"kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.166610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.166576 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.166793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.166642 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.166793 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.166661 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv746\" (UniqueName: \"kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.167241 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.167217 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.168948 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.168928 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.187314 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.187263 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv746\" (UniqueName: \"kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746\") pod \"maas-api-revoke-sub-d576e68b-dgsdg\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.220986 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.220961 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" path="/var/lib/kubelet/pods/f42d55d7-f6ca-4b76-a5e4-e0d954849cbd/volumes" Oct 29 23:15:19.247691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.247669 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:19.541214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:19.541187 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg"] Oct 29 23:15:19.543879 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:15:19.543845 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf012b6fd_3622_4fc4_ae74_55a3e05abd14.slice/crio-a426d5f920b6a2b83e588157dacd1932f02fbfff59520bb12f26384a10b9b28f WatchSource:0}: Error finding container a426d5f920b6a2b83e588157dacd1932f02fbfff59520bb12f26384a10b9b28f: Status 404 returned error can't find the container with id a426d5f920b6a2b83e588157dacd1932f02fbfff59520bb12f26384a10b9b28f Oct 29 23:15:20.243208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:20.243169 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" event={"ID":"f012b6fd-3622-4fc4-ae74-55a3e05abd14","Type":"ContainerStarted","Data":"199797c0328e84ca003feb530a1e6c651d255e6ed48b279a7f8d693ef12eeb0a"} Oct 29 23:15:20.243682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:20.243227 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" event={"ID":"f012b6fd-3622-4fc4-ae74-55a3e05abd14","Type":"ContainerStarted","Data":"a426d5f920b6a2b83e588157dacd1932f02fbfff59520bb12f26384a10b9b28f"} Oct 29 23:15:20.288647 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:20.288565 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" podStartSLOduration=2.288543372 podStartE2EDuration="2.288543372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:20.286793043 +0000 UTC m=+3587.180867965" watchObservedRunningTime="2026-10-29 23:15:20.288543372 +0000 UTC m=+3587.182618282" Oct 29 23:15:21.249848 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:21.249809 5367 generic.go:356] "Generic (PLEG): container finished" podID="f012b6fd-3622-4fc4-ae74-55a3e05abd14" containerID="199797c0328e84ca003feb530a1e6c651d255e6ed48b279a7f8d693ef12eeb0a" exitCode=0 Oct 29 23:15:21.250257 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:21.249900 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" event={"ID":"f012b6fd-3622-4fc4-ae74-55a3e05abd14","Type":"ContainerDied","Data":"199797c0328e84ca003feb530a1e6c651d255e6ed48b279a7f8d693ef12eeb0a"} Oct 29 23:15:22.026250 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.026214 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc"] Oct 29 23:15:22.040055 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.040025 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-482tc"] Oct 29 23:15:22.334308 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:15:22.334253 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:15:22.387691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.387670 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:22.393277 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.393255 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746\" (UniqueName: \"kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746\") pod \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " Oct 29 23:15:22.393376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.393318 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca\") pod \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " Oct 29 23:15:22.393434 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.393373 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca\") pod \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\" (UID: \"f012b6fd-3622-4fc4-ae74-55a3e05abd14\") " Oct 29 23:15:22.393591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.393567 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca" pod "f012b6fd-3622-4fc4-ae74-55a3e05abd14" (UID: "f012b6fd-3622-4fc4-ae74-55a3e05abd14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.395227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.395206 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746" pod "f012b6fd-3622-4fc4-ae74-55a3e05abd14" (UID: "f012b6fd-3622-4fc4-ae74-55a3e05abd14"). InnerVolumeSpecName "kube-api-access-wv746". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:22.395330 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.395249 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca" pod "f012b6fd-3622-4fc4-ae74-55a3e05abd14" (UID: "f012b6fd-3622-4fc4-ae74-55a3e05abd14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:22.494379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.494352 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv746\" (UniqueName: \"kubernetes.io/projected/f012b6fd-3622-4fc4-ae74-55a3e05abd14-kube-api-access-wv746\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.494379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.494376 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.494379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:22.494386 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012b6fd-3622-4fc4-ae74-55a3e05abd14-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:23.219802 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:23.219769 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f189a1-e108-4ce4-86a5-416976ad4961" path="/var/lib/kubelet/pods/85f189a1-e108-4ce4-86a5-416976ad4961/volumes" Oct 29 23:15:23.262171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:23.262150 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" Oct 29 23:15:23.262317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:23.262187 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg" event={"ID":"f012b6fd-3622-4fc4-ae74-55a3e05abd14","Type":"ContainerDied","Data":"a426d5f920b6a2b83e588157dacd1932f02fbfff59520bb12f26384a10b9b28f"} Oct 29 23:15:23.262317 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:23.262229 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a426d5f920b6a2b83e588157dacd1932f02fbfff59520bb12f26384a10b9b28f" Oct 29 23:15:36.352820 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.352785 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx"] Oct 29 23:15:36.353212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.353179 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f012b6fd-3622-4fc4-ae74-55a3e05abd14" containerName="revoke-keys" Oct 29 23:15:36.353212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.353191 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f012b6fd-3622-4fc4-ae74-55a3e05abd14" containerName="revoke-keys" Oct 29 23:15:36.353212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.353205 5367 cpu_manager.go:414] "containerMap: removing container" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:36.353212 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.353210 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:36.353354 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.353280 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f012b6fd-3622-4fc4-ae74-55a3e05abd14" containerName="revoke-keys" Oct 29 23:15:36.353354 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.353290 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42d55d7-f6ca-4b76-a5e4-e0d954849cbd" containerName="cleanup" Oct 29 23:15:36.356810 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.356790 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.387460 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.387433 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:15:36.387858 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.387441 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:15:36.387981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.387525 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:15:36.387981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.387641 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:15:36.389755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.389736 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx"] Oct 29 23:15:36.455512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.455483 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887ct\" (UniqueName: \"kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.455718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.455552 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.455718 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.455633 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.556665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.556629 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.556840 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.556680 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.556840 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.556734 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887ct\" (UniqueName: \"kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.557362 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.557338 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.558927 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.558907 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.578031 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.578001 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887ct\" (UniqueName: \"kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct\") pod \"maas-api-revoke-sub-d2bc93bc-7ljhx\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.669750 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.669660 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:36.964153 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:36.964130 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx"] Oct 29 23:15:36.965957 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:15:36.965932 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c787cee_779c_4070_bc24_260bd7af1f0a.slice/crio-ca2800fb30d954219c6ddbc8d8cba069e91d291b386da76c7ab16b0e4075f65f WatchSource:0}: Error finding container ca2800fb30d954219c6ddbc8d8cba069e91d291b386da76c7ab16b0e4075f65f: Status 404 returned error can't find the container with id ca2800fb30d954219c6ddbc8d8cba069e91d291b386da76c7ab16b0e4075f65f Oct 29 23:15:37.347764 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:37.347723 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" event={"ID":"8c787cee-779c-4070-bc24-260bd7af1f0a","Type":"ContainerStarted","Data":"b43c62a5c4aaac7a5b122fc8590d32f96b38e1eac208ba813e5ff948507a1f34"} Oct 29 23:15:37.347920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:37.347777 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" event={"ID":"8c787cee-779c-4070-bc24-260bd7af1f0a","Type":"ContainerStarted","Data":"ca2800fb30d954219c6ddbc8d8cba069e91d291b386da76c7ab16b0e4075f65f"} Oct 29 23:15:37.390352 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:37.390306 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" podStartSLOduration=1.390292997 podStartE2EDuration="1.390292997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:37.388374307 +0000 UTC m=+3604.282449226" watchObservedRunningTime="2026-10-29 23:15:37.390292997 +0000 UTC m=+3604.284367916" Oct 29 23:15:38.354411 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:38.354370 5367 generic.go:356] "Generic (PLEG): container finished" podID="8c787cee-779c-4070-bc24-260bd7af1f0a" containerID="b43c62a5c4aaac7a5b122fc8590d32f96b38e1eac208ba813e5ff948507a1f34" exitCode=0 Oct 29 23:15:38.354624 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:38.354431 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" event={"ID":"8c787cee-779c-4070-bc24-260bd7af1f0a","Type":"ContainerDied","Data":"b43c62a5c4aaac7a5b122fc8590d32f96b38e1eac208ba813e5ff948507a1f34"} Oct 29 23:15:39.498776 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.498747 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:39.587106 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.587072 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca\") pod \"8c787cee-779c-4070-bc24-260bd7af1f0a\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " Oct 29 23:15:39.587282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.587163 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct\" (UniqueName: \"kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct\") pod \"8c787cee-779c-4070-bc24-260bd7af1f0a\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " Oct 29 23:15:39.587282 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.587196 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca\") pod \"8c787cee-779c-4070-bc24-260bd7af1f0a\" (UID: \"8c787cee-779c-4070-bc24-260bd7af1f0a\") " Oct 29 23:15:39.587621 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.587597 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca" pod "8c787cee-779c-4070-bc24-260bd7af1f0a" (UID: "8c787cee-779c-4070-bc24-260bd7af1f0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:39.589242 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.589221 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca" pod "8c787cee-779c-4070-bc24-260bd7af1f0a" (UID: "8c787cee-779c-4070-bc24-260bd7af1f0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:39.589347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.589327 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct" pod "8c787cee-779c-4070-bc24-260bd7af1f0a" (UID: "8c787cee-779c-4070-bc24-260bd7af1f0a"). InnerVolumeSpecName "kube-api-access-887ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:39.688254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.688174 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887ct\" (UniqueName: \"kubernetes.io/projected/8c787cee-779c-4070-bc24-260bd7af1f0a-kube-api-access-887ct\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:39.688254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.688204 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:39.688254 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:39.688214 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c787cee-779c-4070-bc24-260bd7af1f0a-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:15:40.348779 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:40.348746 5367 scope.go:122] "RemoveContainer" containerID="6291748c031363d602288e8569c5a8cca160be28104bde43daf6a44ca5cdd3c6" Oct 29 23:15:40.357925 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:40.357903 5367 scope.go:122] "RemoveContainer" containerID="98cda1fce56cb126464eb450403dd7407d2452d79784b9e7e31850586097197c" Oct 29 23:15:40.367165 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:40.366998 5367 scope.go:122] "RemoveContainer" containerID="9d6d094cd63446fea5ac8a22c4ceecb71f56fea5eaac870f18d31398ca4acf1e" Oct 29 23:15:40.371724 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:40.371678 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" event={"ID":"8c787cee-779c-4070-bc24-260bd7af1f0a","Type":"ContainerDied","Data":"ca2800fb30d954219c6ddbc8d8cba069e91d291b386da76c7ab16b0e4075f65f"} Oct 29 23:15:40.371832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:40.371746 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2800fb30d954219c6ddbc8d8cba069e91d291b386da76c7ab16b0e4075f65f" Oct 29 23:15:40.371832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:40.371745 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx" Oct 29 23:15:44.027537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:44.027493 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8"] Oct 29 23:15:44.041443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:44.041409 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j7nb8"] Oct 29 23:15:45.219759 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:45.219726 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb7dddf-aef0-4a68-a899-14ad6fbe70d7" path="/var/lib/kubelet/pods/3fb7dddf-aef0-4a68-a899-14ad6fbe70d7/volumes" Oct 29 23:15:48.611777 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.611731 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2"] Oct 29 23:15:48.612224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.612144 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8c787cee-779c-4070-bc24-260bd7af1f0a" containerName="revoke-keys" Oct 29 23:15:48.612224 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.612157 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c787cee-779c-4070-bc24-260bd7af1f0a" containerName="revoke-keys" Oct 29 23:15:48.612320 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.612240 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c787cee-779c-4070-bc24-260bd7af1f0a" containerName="revoke-keys" Oct 29 23:15:48.617218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.617199 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.634427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.634398 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2250ca-openshift-default-dockercfg-vcszm\"" Oct 29 23:15:48.650569 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.650540 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2"] Oct 29 23:15:48.677029 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677002 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677035 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677078 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677191 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677133 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677234 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsh8\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677260 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677317 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677375 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677370 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.677580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.677554 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.778919 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.778885 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.778932 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.778953 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.778995 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779017 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779046 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsh8\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779102 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779154 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779516 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779189 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779813 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779823 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779847 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.779952 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779922 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.780141 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.779930 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.781286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.781263 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.781582 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.781563 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.805265 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.805231 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.805816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.805786 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsh8\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8\") pod \"e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:48.932156 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:48.932056 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:49.260488 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.260459 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2"] Oct 29 23:15:49.262542 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:15:49.262516 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d06498_6530_4eb5_a6d0_999354472678.slice/crio-348ef3a7e4df89ce9d9aecd6269b622e49fb09107d7b0a81f89d2ba46b042cb0 WatchSource:0}: Error finding container 348ef3a7e4df89ce9d9aecd6269b622e49fb09107d7b0a81f89d2ba46b042cb0: Status 404 returned error can't find the container with id 348ef3a7e4df89ce9d9aecd6269b622e49fb09107d7b0a81f89d2ba46b042cb0 Oct 29 23:15:49.265116 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.265083 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:15:49.265219 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.265158 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:15:49.265219 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.265199 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:15:49.422368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.422335 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" event={"ID":"12d06498-6530-4eb5-a6d0-999354472678","Type":"ContainerStarted","Data":"d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1"} Oct 29 23:15:49.422605 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.422392 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" event={"ID":"12d06498-6530-4eb5-a6d0-999354472678","Type":"ContainerStarted","Data":"348ef3a7e4df89ce9d9aecd6269b622e49fb09107d7b0a81f89d2ba46b042cb0"} Oct 29 23:15:49.471671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.471584 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podStartSLOduration=1.471565194 podStartE2EDuration="1.471565194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:49.468813502 +0000 UTC m=+3616.362888425" watchObservedRunningTime="2026-10-29 23:15:49.471565194 +0000 UTC m=+3616.365640117" Oct 29 23:15:49.932668 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:49.932628 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:50.937545 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:50.937514 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:51.432580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:51.432548 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:51.433512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:51.433492 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:15:52.331218 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:15:52.331180 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:15:53.275205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.275167 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2"] Oct 29 23:15:53.278815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.278791 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.313983 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.313944 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:15:53.313983 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.313944 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:15:53.313983 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.313969 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2250ca-serving-cert\"" Oct 29 23:15:53.314959 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.314919 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 23:15:53.315171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.315147 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 23:15:53.317431 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.317408 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2"] Oct 29 23:15:53.428964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.428936 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.429338 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.428988 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.429338 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.429019 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94hw\" (UniqueName: \"kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.429338 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.429111 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.530151 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.530077 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.530328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.530200 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.530328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.530233 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94hw\" (UniqueName: \"kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.530328 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.530265 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.530920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.530894 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.532862 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.532839 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.532981 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.532921 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.552430 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.552405 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94hw\" (UniqueName: \"kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw\") pod \"maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.592386 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.592361 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:53.894285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:53.894261 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2"] Oct 29 23:15:53.896093 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:15:53.896064 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d047b95_d5b2_4de6_9e32_19673d9dd5fa.slice/crio-9348f61d4bf55eb284431676b38e2905e722e74e1f17669db7831e4c84aa3ea1 WatchSource:0}: Error finding container 9348f61d4bf55eb284431676b38e2905e722e74e1f17669db7831e4c84aa3ea1: Status 404 returned error can't find the container with id 9348f61d4bf55eb284431676b38e2905e722e74e1f17669db7831e4c84aa3ea1 Oct 29 23:15:54.451376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:54.451336 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" event={"ID":"7d047b95-d5b2-4de6-9e32-19673d9dd5fa","Type":"ContainerStarted","Data":"7c9deaf6d9d9f0e55f7531aba5194773de4b5ab2ee39751e370ef2843e9f7971"} Oct 29 23:15:54.451843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:54.451404 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" event={"ID":"7d047b95-d5b2-4de6-9e32-19673d9dd5fa","Type":"ContainerStarted","Data":"9348f61d4bf55eb284431676b38e2905e722e74e1f17669db7831e4c84aa3ea1"} Oct 29 23:15:54.451843 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:54.451430 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:15:54.498681 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:15:54.498573 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" podStartSLOduration=1.171301061 podStartE2EDuration="1.498555855s" totalImagesPullingTime="327.254794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:54.497625292 +0000 UTC m=+3621.391700211" watchObservedRunningTime="2026-10-29 23:15:54.498555855 +0000 UTC m=+3621.392630778" Oct 29 23:16:00.482468 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:00.482418 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:16:04.027557 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:04.027519 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht"] Oct 29 23:16:04.041761 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:04.041724 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8trht"] Oct 29 23:16:05.220152 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:05.220113 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ade951-6c15-460c-b543-7a43cb43aabc" path="/var/lib/kubelet/pods/91ade951-6c15-460c-b543-7a43cb43aabc/volumes" Oct 29 23:16:06.434674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.434592 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" start-of-body= Oct 29 23:16:06.434674 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.434662 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" Oct 29 23:16:06.527377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.527344 5367 generic.go:356] "Generic (PLEG): container finished" podID="12d06498-6530-4eb5-a6d0-999354472678" containerID="d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1" exitCode=0 Oct 29 23:16:06.527541 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.527413 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" event={"ID":"12d06498-6530-4eb5-a6d0-999354472678","Type":"ContainerDied","Data":"d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1"} Oct 29 23:16:06.527792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.527779 5367 scope.go:122] "RemoveContainer" containerID="d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1" Oct 29 23:16:06.528895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.528865 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:16:06.529035 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.528946 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:16:06.529035 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:06.528993 5367 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:16:07.535450 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:07.535418 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" event={"ID":"12d06498-6530-4eb5-a6d0-999354472678","Type":"ContainerStarted","Data":"9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348"} Oct 29 23:16:07.933038 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:07.932998 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:16:07.933227 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:07.933196 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" start-of-body= Oct 29 23:16:07.933273 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:07.933243 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" Oct 29 23:16:08.932980 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:08.932947 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" start-of-body= Oct 29 23:16:08.933390 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:08.933000 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" Oct 29 23:16:09.932744 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:09.932692 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" start-of-body= Oct 29 23:16:09.932933 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:09.932773 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" Oct 29 23:16:10.191305 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.190342 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh"] Oct 29 23:16:10.198148 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.197866 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.210478 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.210434 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.210693 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.210673 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.210941 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.210917 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.211041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.211025 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.211132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.211118 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.211205 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.211188 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbc9f\" (UniqueName: \"kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.214120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.214091 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh"] Oct 29 23:16:10.224831 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.224812 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2250ca\"/\"default-dockercfg-pzktt\"" Oct 29 23:16:10.224994 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.224975 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2250ca\"/\"openshift-service-ca.crt\"" Oct 29 23:16:10.225087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.225047 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2250ca\"/\"simulator-single-main-2250ca-kserve-self-signed-certs\"" Oct 29 23:16:10.228246 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.228056 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2250ca\"/\"kube-root-ca.crt\"" Oct 29 23:16:10.311965 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.311926 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.311977 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312007 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312063 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312154 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312108 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312170 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbc9f\" (UniqueName: \"kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312544 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312521 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312635 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.312863 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.312842 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.314354 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.314335 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.314667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.314651 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.337841 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.337822 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbc9f\" (UniqueName: \"kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f\") pod \"simulator-single-main-2250ca-kserve-6556d78c87-9djlh\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.510591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.510509 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:10.838672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.838646 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh"] Oct 29 23:16:10.840281 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:16:10.840244 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b5b6b65_84f5_41ac_a97c_f63cf553e88d.slice/crio-72d667cffa83523580523af58dd0495826530abda7e9e452d408350662edaf99 WatchSource:0}: Error finding container 72d667cffa83523580523af58dd0495826530abda7e9e452d408350662edaf99: Status 404 returned error can't find the container with id 72d667cffa83523580523af58dd0495826530abda7e9e452d408350662edaf99 Oct 29 23:16:10.932954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.932929 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" start-of-body= Oct 29 23:16:10.933095 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:10.932968 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" Oct 29 23:16:11.580403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:11.580361 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" event={"ID":"6b5b6b65-84f5-41ac-a97c-f63cf553e88d","Type":"ContainerStarted","Data":"c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2"} Oct 29 23:16:11.580403 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:11.580406 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" event={"ID":"6b5b6b65-84f5-41ac-a97c-f63cf553e88d","Type":"ContainerStarted","Data":"72d667cffa83523580523af58dd0495826530abda7e9e452d408350662edaf99"} Oct 29 23:16:11.625861 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:11.625813 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" podStartSLOduration=1.625800728 podStartE2EDuration="1.625800728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:11.624548213 +0000 UTC m=+3638.518623125" watchObservedRunningTime="2026-10-29 23:16:11.625800728 +0000 UTC m=+3638.519875689" Oct 29 23:16:11.932576 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:11.932495 5367 patch_prober.go:29] interesting pod/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" start-of-body= Oct 29 23:16:11.932576 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:11.932548 5367 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.156:15021/healthz/ready\": dial tcp 10.133.1.156:15021: connect: connection refused" Oct 29 23:16:12.510922 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:12.510875 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:13.191449 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:13.191416 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:16:13.191957 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:13.191919 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:16:13.192756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:13.192734 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:16:20.511293 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:20.511258 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:20.523721 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:20.523677 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:20.637552 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:20.637525 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:22.026117 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:22.026080 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm"] Oct 29 23:16:22.039622 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:22.039596 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjmjm"] Oct 29 23:16:22.330140 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:16:22.330101 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:16:23.219711 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:23.219675 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf149d55-ae3f-4c95-b720-4f6b1b74f7e5" path="/var/lib/kubelet/pods/bf149d55-ae3f-4c95-b720-4f6b1b74f7e5/volumes" Oct 29 23:16:40.410195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:40.410161 5367 scope.go:122] "RemoveContainer" containerID="9f59e4e1a993a936f807f71100c80fe7e76ffce9eaa8599cf9f3d50b470ead9a" Oct 29 23:16:40.419025 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:40.419000 5367 scope.go:122] "RemoveContainer" containerID="2489ecd19c83c4d106c375e2a92445f97ec0e07ae04a93cbf14202db89b883cf" Oct 29 23:16:40.426960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:40.426938 5367 scope.go:122] "RemoveContainer" containerID="2e4078ac2101d8685d0c1a80a460821c528e13b92cac3ac42c205b73f566aa3e" Oct 29 23:16:42.224592 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:42.224559 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh"] Oct 29 23:16:50.613769 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:50.613724 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2"] Oct 29 23:16:50.614296 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:50.613992 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" podUID="7d047b95-d5b2-4de6-9e32-19673d9dd5fa" containerName="maas-api" containerID="cri-o://7c9deaf6d9d9f0e55f7531aba5194773de4b5ab2ee39751e370ef2843e9f7971" gracePeriod=30 Oct 29 23:16:51.803353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.803304 5367 generic.go:356] "Generic (PLEG): container finished" podID="7d047b95-d5b2-4de6-9e32-19673d9dd5fa" containerID="7c9deaf6d9d9f0e55f7531aba5194773de4b5ab2ee39751e370ef2843e9f7971" exitCode=0 Oct 29 23:16:51.803353 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.803327 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" event={"ID":"7d047b95-d5b2-4de6-9e32-19673d9dd5fa","Type":"ContainerDied","Data":"7c9deaf6d9d9f0e55f7531aba5194773de4b5ab2ee39751e370ef2843e9f7971"} Oct 29 23:16:51.881102 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.881074 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:16:51.922436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.922408 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw\" (UniqueName: \"kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw\") pod \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " Oct 29 23:16:51.922636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.922454 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca\") pod \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " Oct 29 23:16:51.922636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.922480 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs\") pod \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " Oct 29 23:16:51.922636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.922540 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls\") pod \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\" (UID: \"7d047b95-d5b2-4de6-9e32-19673d9dd5fa\") " Oct 29 23:16:51.922887 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.922859 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca" pod "7d047b95-d5b2-4de6-9e32-19673d9dd5fa" (UID: "7d047b95-d5b2-4de6-9e32-19673d9dd5fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:51.924626 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.924602 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls" pod "7d047b95-d5b2-4de6-9e32-19673d9dd5fa" (UID: "7d047b95-d5b2-4de6-9e32-19673d9dd5fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:51.924725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.924686 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw" pod "7d047b95-d5b2-4de6-9e32-19673d9dd5fa" (UID: "7d047b95-d5b2-4de6-9e32-19673d9dd5fa"). InnerVolumeSpecName "kube-api-access-m94hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:51.924789 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:51.924722 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs" pod "7d047b95-d5b2-4de6-9e32-19673d9dd5fa" (UID: "7d047b95-d5b2-4de6-9e32-19673d9dd5fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:52.023432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.023347 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94hw\" (UniqueName: \"kubernetes.io/projected/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-kube-api-access-m94hw\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:52.023432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.023379 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:52.023432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.023390 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:52.023432 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.023398 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d047b95-d5b2-4de6-9e32-19673d9dd5fa-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:52.331879 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:16:52.331844 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:16:52.810537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.810508 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" Oct 29 23:16:52.811040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.810547 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2" event={"ID":"7d047b95-d5b2-4de6-9e32-19673d9dd5fa","Type":"ContainerDied","Data":"9348f61d4bf55eb284431676b38e2905e722e74e1f17669db7831e4c84aa3ea1"} Oct 29 23:16:52.811040 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.810617 5367 scope.go:122] "RemoveContainer" containerID="7c9deaf6d9d9f0e55f7531aba5194773de4b5ab2ee39751e370ef2843e9f7971" Oct 29 23:16:52.872271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.872237 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2"] Oct 29 23:16:52.885251 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:52.885219 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2250ca-cdb4c8968-bf5g2"] Oct 29 23:16:53.222757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:53.222638 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d047b95-d5b2-4de6-9e32-19673d9dd5fa" path="/var/lib/kubelet/pods/7d047b95-d5b2-4de6-9e32-19673d9dd5fa/volumes" Oct 29 23:16:57.245500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.245440 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" podUID="6b5b6b65-84f5-41ac-a97c-f63cf553e88d" containerName="main" containerID="cri-o://c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2" gracePeriod=45 Oct 29 23:16:57.490304 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.490280 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:57.571636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.571610 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm\") pod \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " Oct 29 23:16:57.571816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.571682 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs\") pod \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " Oct 29 23:16:57.571816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.571756 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache\") pod \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " Oct 29 23:16:57.571816 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.571805 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home\") pod \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " Oct 29 23:16:57.572000 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.571895 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f\" (UniqueName: \"kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f\") pod \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " Oct 29 23:16:57.572000 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.571949 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir\") pod \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\" (UID: \"6b5b6b65-84f5-41ac-a97c-f63cf553e88d\") " Oct 29 23:16:57.572128 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.572054 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache" pod "6b5b6b65-84f5-41ac-a97c-f63cf553e88d" (UID: "6b5b6b65-84f5-41ac-a97c-f63cf553e88d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:57.572128 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.572068 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home" pod "6b5b6b65-84f5-41ac-a97c-f63cf553e88d" (UID: "6b5b6b65-84f5-41ac-a97c-f63cf553e88d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:57.572286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.572268 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir" pod "6b5b6b65-84f5-41ac-a97c-f63cf553e88d" (UID: "6b5b6b65-84f5-41ac-a97c-f63cf553e88d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:57.572348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.572311 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:57.572348 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.572334 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:57.574271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.574244 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm" pod "6b5b6b65-84f5-41ac-a97c-f63cf553e88d" (UID: "6b5b6b65-84f5-41ac-a97c-f63cf553e88d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:57.574418 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.574290 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f" pod "6b5b6b65-84f5-41ac-a97c-f63cf553e88d" (UID: "6b5b6b65-84f5-41ac-a97c-f63cf553e88d"). InnerVolumeSpecName "kube-api-access-lbc9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:57.577120 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.577090 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs" pod "6b5b6b65-84f5-41ac-a97c-f63cf553e88d" (UID: "6b5b6b65-84f5-41ac-a97c-f63cf553e88d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:57.673801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.673756 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:57.673801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.673792 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:57.673801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.673802 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:57.673801 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.673813 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbc9f\" (UniqueName: \"kubernetes.io/projected/6b5b6b65-84f5-41ac-a97c-f63cf553e88d-kube-api-access-lbc9f\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:16:57.839504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.839421 5367 generic.go:356] "Generic (PLEG): container finished" podID="6b5b6b65-84f5-41ac-a97c-f63cf553e88d" containerID="c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2" exitCode=0 Oct 29 23:16:57.839504 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.839465 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" Oct 29 23:16:57.839712 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.839507 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" event={"ID":"6b5b6b65-84f5-41ac-a97c-f63cf553e88d","Type":"ContainerDied","Data":"c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2"} Oct 29 23:16:57.839712 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.839552 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh" event={"ID":"6b5b6b65-84f5-41ac-a97c-f63cf553e88d","Type":"ContainerDied","Data":"72d667cffa83523580523af58dd0495826530abda7e9e452d408350662edaf99"} Oct 29 23:16:57.839712 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.839569 5367 scope.go:122] "RemoveContainer" containerID="c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2" Oct 29 23:16:57.848996 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.848976 5367 scope.go:122] "RemoveContainer" containerID="c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2" Oct 29 23:16:57.849259 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:16:57.849241 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2\": container with ID starting with c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2 not found: ID does not exist" containerID="c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2" Oct 29 23:16:57.849306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.849268 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2"} err="failed to get container status \"c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2\": rpc error: code = NotFound desc = could not find container \"c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2\": container with ID starting with c84ced31af4b7d5e0070f0be585a590bc25754044dfda93394d9af343dca4cf2 not found: ID does not exist" Oct 29 23:16:57.907496 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.907473 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh"] Oct 29 23:16:57.913126 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:57.913107 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2250ca/simulator-single-main-2250ca-kserve-6556d78c87-9djlh"] Oct 29 23:16:59.220038 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:59.220002 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5b6b65-84f5-41ac-a97c-f63cf553e88d" path="/var/lib/kubelet/pods/6b5b6b65-84f5-41ac-a97c-f63cf553e88d/volumes" Oct 29 23:16:59.931964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:59.931930 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2"] Oct 29 23:16:59.932174 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:16:59.932156 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" containerID="cri-o://9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348" gracePeriod=30 Oct 29 23:17:05.139822 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.139789 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98"] Oct 29 23:17:05.140399 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.140377 5367 cpu_manager.go:414] "containerMap: removing container" podUID="6b5b6b65-84f5-41ac-a97c-f63cf553e88d" containerName="main" Oct 29 23:17:05.140457 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.140404 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5b6b65-84f5-41ac-a97c-f63cf553e88d" containerName="main" Oct 29 23:17:05.140500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.140457 5367 cpu_manager.go:414] "containerMap: removing container" podUID="7d047b95-d5b2-4de6-9e32-19673d9dd5fa" containerName="maas-api" Oct 29 23:17:05.140500 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.140468 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d047b95-d5b2-4de6-9e32-19673d9dd5fa" containerName="maas-api" Oct 29 23:17:05.140596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.140581 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5b6b65-84f5-41ac-a97c-f63cf553e88d" containerName="main" Oct 29 23:17:05.140632 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.140612 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d047b95-d5b2-4de6-9e32-19673d9dd5fa" containerName="maas-api" Oct 29 23:17:05.144424 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.144395 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.164484 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.164437 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e33378d4-kserve-self-signed-certs\"" Oct 29 23:17:05.181641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.181610 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98"] Oct 29 23:17:05.190354 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.190333 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:17:05.246111 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246051 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.246111 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246098 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.246303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246126 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.246303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246186 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.246303 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246232 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246322 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246337 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246362 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246420 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246494 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket\") pod \"12d06498-6530-4eb5-a6d0-999354472678\" (UID: \"12d06498-6530-4eb5-a6d0-999354472678\") " Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246589 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246627 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246683 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246753 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246821 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.246929 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247091 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247152 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.247286 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247234 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7bsp\" (UniqueName: \"kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.248291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247301 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.248291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247426 5367 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-data\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.248291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247447 5367 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.248291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247460 5367 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-credential-socket\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.248291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247475 5367 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-workload-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.248291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.247499 5367 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d06498-6530-4eb5-a6d0-999354472678-istiod-ca-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.249283 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.249240 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:05.249543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.249522 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "kube-api-access-gbsh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:05.249678 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.249590 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:17:05.251756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.251719 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy" pod "12d06498-6530-4eb5-a6d0-999354472678" (UID: "12d06498-6530-4eb5-a6d0-999354472678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:05.348124 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348093 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348140 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348173 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348239 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348326 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348283 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348405 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7bsp\" (UniqueName: \"kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348510 5367 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.348543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348528 5367 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d06498-6530-4eb5-a6d0-999354472678-istio-podinfo\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.348543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348540 5367 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d06498-6530-4eb5-a6d0-999354472678-istio-envoy\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.348805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348553 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbsh8\" (UniqueName: \"kubernetes.io/projected/12d06498-6530-4eb5-a6d0-999354472678-kube-api-access-gbsh8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:05.348805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348654 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348684 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.348805 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.348759 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.350621 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.350599 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.350791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.350738 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.371738 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.371691 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7bsp\" (UniqueName: \"kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp\") pod \"e2e-same-id-a-e33378d4-kserve-859d798777-czq98\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.458652 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.458618 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:05.768676 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.768570 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98"] Oct 29 23:17:05.773072 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:17:05.773045 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7d37349_b6d7_4a79_9c9c_bee9ba40422b.slice/crio-1707657a01ae0fb145362669cee0026539947ba4e326af54573d33e2324028d9 WatchSource:0}: Error finding container 1707657a01ae0fb145362669cee0026539947ba4e326af54573d33e2324028d9: Status 404 returned error can't find the container with id 1707657a01ae0fb145362669cee0026539947ba4e326af54573d33e2324028d9 Oct 29 23:17:05.891285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.891254 5367 generic.go:356] "Generic (PLEG): container finished" podID="12d06498-6530-4eb5-a6d0-999354472678" containerID="9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348" exitCode=0 Oct 29 23:17:05.891482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.891304 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" Oct 29 23:17:05.891482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.891345 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" event={"ID":"12d06498-6530-4eb5-a6d0-999354472678","Type":"ContainerDied","Data":"9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348"} Oct 29 23:17:05.891482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.891414 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2" event={"ID":"12d06498-6530-4eb5-a6d0-999354472678","Type":"ContainerDied","Data":"348ef3a7e4df89ce9d9aecd6269b622e49fb09107d7b0a81f89d2ba46b042cb0"} Oct 29 23:17:05.891482 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.891444 5367 scope.go:122] "RemoveContainer" containerID="9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348" Oct 29 23:17:05.893110 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.893085 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" event={"ID":"b7d37349-b6d7-4a79-9c9c-bee9ba40422b","Type":"ContainerStarted","Data":"a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755"} Oct 29 23:17:05.893223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.893121 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" event={"ID":"b7d37349-b6d7-4a79-9c9c-bee9ba40422b","Type":"ContainerStarted","Data":"1707657a01ae0fb145362669cee0026539947ba4e326af54573d33e2324028d9"} Oct 29 23:17:05.901543 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.901526 5367 scope.go:122] "RemoveContainer" containerID="d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1" Oct 29 23:17:05.909631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.909613 5367 scope.go:122] "RemoveContainer" containerID="9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348" Oct 29 23:17:05.909956 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:17:05.909937 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348\": container with ID starting with 9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348 not found: ID does not exist" containerID="9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348" Oct 29 23:17:05.910041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.909968 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348"} err="failed to get container status \"9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348\": rpc error: code = NotFound desc = could not find container \"9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348\": container with ID starting with 9edc09c2e86ad05054075eca5f210e40e5293e7915ad089ee69bb27bd85f6348 not found: ID does not exist" Oct 29 23:17:05.910041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.909999 5367 scope.go:122] "RemoveContainer" containerID="d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1" Oct 29 23:17:05.910259 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:17:05.910243 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1\": container with ID starting with d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1 not found: ID does not exist" containerID="d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1" Oct 29 23:17:05.910300 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.910265 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1"} err="failed to get container status \"d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1\": rpc error: code = NotFound desc = could not find container \"d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1\": container with ID starting with d96ca89d4d1474540864d3ad0d6243ab68f67ad326ddb1c325eb96179c079de1 not found: ID does not exist" Oct 29 23:17:05.955249 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.955220 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2"] Oct 29 23:17:05.968258 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:05.968236 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2250ca-openshift-default-57885cfb5cct2f2"] Oct 29 23:17:06.003565 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:06.003521 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" podStartSLOduration=1.003509 podStartE2EDuration="1.003509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:06.00273992 +0000 UTC m=+3692.896814841" watchObservedRunningTime="2026-10-29 23:17:06.003509 +0000 UTC m=+3692.897583921" Oct 29 23:17:06.459171 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:06.459137 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:07.222223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:07.222192 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d06498-6530-4eb5-a6d0-999354472678" path="/var/lib/kubelet/pods/12d06498-6530-4eb5-a6d0-999354472678/volumes" Oct 29 23:17:09.832939 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:09.832889 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:09.833444 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:09.832963 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:15.459204 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:15.459099 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:15.471810 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:15.471778 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:15.990232 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:15.990200 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:17:22.331553 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:17:22.331513 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:17:27.874372 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874334 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q"] Oct 29 23:17:27.874913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874741 5367 cpu_manager.go:414] "containerMap: removing container" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" Oct 29 23:17:27.874913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874755 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" Oct 29 23:17:27.874913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874771 5367 cpu_manager.go:414] "containerMap: removing container" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" Oct 29 23:17:27.874913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874779 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" Oct 29 23:17:27.874913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874843 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" Oct 29 23:17:27.874913 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.874852 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d06498-6530-4eb5-a6d0-999354472678" containerName="istio-proxy" Oct 29 23:17:27.878086 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.878060 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:27.911896 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.911843 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e33378d4-kserve-self-signed-certs\"" Oct 29 23:17:27.913920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:27.913896 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q"] Oct 29 23:17:28.001779 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.001745 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.001779 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.001786 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddwg\" (UniqueName: \"kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.002024 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.001877 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.002024 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.001922 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.002024 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.002007 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.002140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.002054 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.103512 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.103463 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.103743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.103558 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.103743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.103594 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddwg\" (UniqueName: \"kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.103743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.103658 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.103743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.103732 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.104011 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.103785 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.104074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.104023 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.104115 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.104072 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.104180 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.104157 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.106074 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.106050 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.106377 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.106349 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.134058 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.133983 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddwg\" (UniqueName: \"kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg\") pod \"e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.192379 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.192339 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:28.490628 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:28.490599 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q"] Oct 29 23:17:28.492558 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:17:28.492534 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18904c37_7555_450b_b8f8_95e7bfb98aa7.slice/crio-68fd3f19b9cc0a8ecf0a60601ec5af6489e4ecf492ac7184cffdc571220bdd1b WatchSource:0}: Error finding container 68fd3f19b9cc0a8ecf0a60601ec5af6489e4ecf492ac7184cffdc571220bdd1b: Status 404 returned error can't find the container with id 68fd3f19b9cc0a8ecf0a60601ec5af6489e4ecf492ac7184cffdc571220bdd1b Oct 29 23:17:29.059464 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:29.059420 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" event={"ID":"18904c37-7555-450b-b8f8-95e7bfb98aa7","Type":"ContainerStarted","Data":"d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b"} Oct 29 23:17:29.059464 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:29.059467 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" event={"ID":"18904c37-7555-450b-b8f8-95e7bfb98aa7","Type":"ContainerStarted","Data":"68fd3f19b9cc0a8ecf0a60601ec5af6489e4ecf492ac7184cffdc571220bdd1b"} Oct 29 23:17:29.104132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:29.104078 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" podStartSLOduration=2.104063824 podStartE2EDuration="2.104063824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:29.102583656 +0000 UTC m=+3715.996658577" watchObservedRunningTime="2026-10-29 23:17:29.104063824 +0000 UTC m=+3715.998138745" Oct 29 23:17:29.193440 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:29.193402 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:38.193390 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:38.193355 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:38.205962 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:38.205938 5367 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:39.162734 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:39.162691 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:17:40.462146 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:40.462110 5367 scope.go:122] "RemoveContainer" containerID="bfcb315f9cc2ef8b530bc71b8e92a553a8b9c24a6d034d737fcd4fb38d206fbc" Oct 29 23:17:40.471063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:40.471043 5367 scope.go:122] "RemoveContainer" containerID="b99dcb4a62990823c49fd211a5ce68eb82c9acb313764734195e1355d9fe809e" Oct 29 23:17:40.478892 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:40.478873 5367 scope.go:122] "RemoveContainer" containerID="78ba611a75a547938db073c8357fdd857db1d745f28a021f46cefb1f2f3999d6" Oct 29 23:17:40.486220 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:40.486204 5367 scope.go:122] "RemoveContainer" containerID="0f021c693335188a150da001f17cd290e1b14ff2676641615446356105d946bf" Oct 29 23:17:43.293769 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.293735 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2"] Oct 29 23:17:43.298790 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.298774 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.318928 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.318904 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2"] Oct 29 23:17:43.330255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.330231 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:17:43.330255 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.330235 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:17:43.330436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.330274 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:17:43.332086 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.332072 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:17:43.385004 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.384977 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.385185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.385029 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.385185 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.385132 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgjm\" (UniqueName: \"kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.486179 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.486147 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.486340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.486188 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgjm\" (UniqueName: \"kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.486340 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.486262 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.486821 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.486798 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.488529 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.488509 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.507436 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.507413 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgjm\" (UniqueName: \"kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm\") pod \"maas-api-revoke-sub-452b6d7b-l9gf2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.611310 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.611286 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:43.911964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:43.911938 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2"] Oct 29 23:17:43.913727 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:17:43.913678 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991f4492_f12a_4b6f_9a9e_d8b4321809e2.slice/crio-bdc3d6ceeed5ad3599532d78ef4acd5e0ec927ebb172436d81549a5ae8320f43 WatchSource:0}: Error finding container bdc3d6ceeed5ad3599532d78ef4acd5e0ec927ebb172436d81549a5ae8320f43: Status 404 returned error can't find the container with id bdc3d6ceeed5ad3599532d78ef4acd5e0ec927ebb172436d81549a5ae8320f43 Oct 29 23:17:44.181064 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:44.181029 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" event={"ID":"991f4492-f12a-4b6f-9a9e-d8b4321809e2","Type":"ContainerStarted","Data":"256375e9a3b15b786b91f732588bbabcb87e35b6e1c615c10ec85d3a85e4e071"} Oct 29 23:17:44.181184 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:44.181115 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" event={"ID":"991f4492-f12a-4b6f-9a9e-d8b4321809e2","Type":"ContainerStarted","Data":"bdc3d6ceeed5ad3599532d78ef4acd5e0ec927ebb172436d81549a5ae8320f43"} Oct 29 23:17:44.226548 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:44.226491 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" podStartSLOduration=1.226477686 podStartE2EDuration="1.226477686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:44.224771839 +0000 UTC m=+3731.118846762" watchObservedRunningTime="2026-10-29 23:17:44.226477686 +0000 UTC m=+3731.120552606" Oct 29 23:17:45.217287 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:45.217249 5367 generic.go:356] "Generic (PLEG): container finished" podID="991f4492-f12a-4b6f-9a9e-d8b4321809e2" containerID="256375e9a3b15b786b91f732588bbabcb87e35b6e1c615c10ec85d3a85e4e071" exitCode=0 Oct 29 23:17:45.218023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:45.217995 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" event={"ID":"991f4492-f12a-4b6f-9a9e-d8b4321809e2","Type":"ContainerDied","Data":"256375e9a3b15b786b91f732588bbabcb87e35b6e1c615c10ec85d3a85e4e071"} Oct 29 23:17:46.351903 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.351863 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:46.352274 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.351914 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:46.619954 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.619915 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:46.744169 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.744123 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca\") pod \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " Oct 29 23:17:46.744361 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.744295 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca\") pod \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " Oct 29 23:17:46.744361 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.744338 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm\" (UniqueName: \"kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm\") pod \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\" (UID: \"991f4492-f12a-4b6f-9a9e-d8b4321809e2\") " Oct 29 23:17:46.744597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.744560 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca" pod "991f4492-f12a-4b6f-9a9e-d8b4321809e2" (UID: "991f4492-f12a-4b6f-9a9e-d8b4321809e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:46.744737 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.744724 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:46.746610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.746581 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca" pod "991f4492-f12a-4b6f-9a9e-d8b4321809e2" (UID: "991f4492-f12a-4b6f-9a9e-d8b4321809e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:17:46.746755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.746633 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm" pod "991f4492-f12a-4b6f-9a9e-d8b4321809e2" (UID: "991f4492-f12a-4b6f-9a9e-d8b4321809e2"). InnerVolumeSpecName "kube-api-access-8jgjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:46.846217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.846181 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991f4492-f12a-4b6f-9a9e-d8b4321809e2-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:46.846217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:46.846214 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgjm\" (UniqueName: \"kubernetes.io/projected/991f4492-f12a-4b6f-9a9e-d8b4321809e2-kube-api-access-8jgjm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:17:47.241149 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:47.241057 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" event={"ID":"991f4492-f12a-4b6f-9a9e-d8b4321809e2","Type":"ContainerDied","Data":"bdc3d6ceeed5ad3599532d78ef4acd5e0ec927ebb172436d81549a5ae8320f43"} Oct 29 23:17:47.241149 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:47.241101 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l9gf2" Oct 29 23:17:47.241149 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:47.241115 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc3d6ceeed5ad3599532d78ef4acd5e0ec927ebb172436d81549a5ae8320f43" Oct 29 23:17:52.330126 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:17:52.330090 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:17:54.479762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:54.479720 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98"] Oct 29 23:17:56.709208 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:56.709166 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q"] Oct 29 23:17:59.291200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:59.290800 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:59.291200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:17:59.290887 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx" podUID="2bcf97eb-0392-4819-8114-e0c06d076ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:18:00.206832 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.206796 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp"] Oct 29 23:18:00.207202 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.207190 5367 cpu_manager.go:414] "containerMap: removing container" podUID="991f4492-f12a-4b6f-9a9e-d8b4321809e2" containerName="revoke-keys" Oct 29 23:18:00.207256 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.207203 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991f4492-f12a-4b6f-9a9e-d8b4321809e2" containerName="revoke-keys" Oct 29 23:18:00.207298 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.207270 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="991f4492-f12a-4b6f-9a9e-d8b4321809e2" containerName="revoke-keys" Oct 29 23:18:00.210537 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.210516 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.231104 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.231071 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp"] Oct 29 23:18:00.242239 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.242191 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:18:00.242239 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.242222 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:18:00.242239 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.242223 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:18:00.242556 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.242227 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:18:00.311864 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.311835 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.312248 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.311888 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.312248 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.311999 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lvd\" (UniqueName: \"kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.413222 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.413183 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.413413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.413244 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.413413 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.413281 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lvd\" (UniqueName: \"kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.413932 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.413907 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.415596 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.415572 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.434665 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.434637 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lvd\" (UniqueName: \"kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd\") pod \"maas-api-revoke-sub-3e31a506-gsdmp\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.524608 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.524524 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:00.831198 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:00.831171 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp"] Oct 29 23:18:00.832684 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:18:00.832657 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a8c416_4abb_4cd5_9e8b_f30707700d54.slice/crio-385b7517dffb491d047d7fe7f7e5f08cec02a0201d432115834b381b2ebecca6 WatchSource:0}: Error finding container 385b7517dffb491d047d7fe7f7e5f08cec02a0201d432115834b381b2ebecca6: Status 404 returned error can't find the container with id 385b7517dffb491d047d7fe7f7e5f08cec02a0201d432115834b381b2ebecca6 Oct 29 23:18:01.337563 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:01.337522 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" event={"ID":"06a8c416-4abb-4cd5-9e8b-f30707700d54","Type":"ContainerStarted","Data":"6e05d894b986b907de89a120e3a6eb6d86fa89ecafcc1c295bf8b172bfd784e2"} Oct 29 23:18:01.338076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:01.337601 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" event={"ID":"06a8c416-4abb-4cd5-9e8b-f30707700d54","Type":"ContainerStarted","Data":"385b7517dffb491d047d7fe7f7e5f08cec02a0201d432115834b381b2ebecca6"} Oct 29 23:18:01.383262 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:01.383204 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" podStartSLOduration=1.383188892 podStartE2EDuration="1.383188892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:01.381098357 +0000 UTC m=+3748.275173279" watchObservedRunningTime="2026-10-29 23:18:01.383188892 +0000 UTC m=+3748.277263813" Oct 29 23:18:02.344355 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:02.344320 5367 generic.go:356] "Generic (PLEG): container finished" podID="06a8c416-4abb-4cd5-9e8b-f30707700d54" containerID="6e05d894b986b907de89a120e3a6eb6d86fa89ecafcc1c295bf8b172bfd784e2" exitCode=0 Oct 29 23:18:02.344787 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:02.344363 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" event={"ID":"06a8c416-4abb-4cd5-9e8b-f30707700d54","Type":"ContainerDied","Data":"6e05d894b986b907de89a120e3a6eb6d86fa89ecafcc1c295bf8b172bfd784e2"} Oct 29 23:18:03.581591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.581561 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:03.655583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.655509 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca\") pod \"06a8c416-4abb-4cd5-9e8b-f30707700d54\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " Oct 29 23:18:03.655583 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.655549 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd\" (UniqueName: \"kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd\") pod \"06a8c416-4abb-4cd5-9e8b-f30707700d54\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " Oct 29 23:18:03.655853 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.655613 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca\") pod \"06a8c416-4abb-4cd5-9e8b-f30707700d54\" (UID: \"06a8c416-4abb-4cd5-9e8b-f30707700d54\") " Oct 29 23:18:03.655983 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.655873 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca" pod "06a8c416-4abb-4cd5-9e8b-f30707700d54" (UID: "06a8c416-4abb-4cd5-9e8b-f30707700d54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:03.657742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.657714 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca" pod "06a8c416-4abb-4cd5-9e8b-f30707700d54" (UID: "06a8c416-4abb-4cd5-9e8b-f30707700d54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:03.657859 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.657794 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd" pod "06a8c416-4abb-4cd5-9e8b-f30707700d54" (UID: "06a8c416-4abb-4cd5-9e8b-f30707700d54"). InnerVolumeSpecName "kube-api-access-95lvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:03.756749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.756692 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:03.756749 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.756744 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a8c416-4abb-4cd5-9e8b-f30707700d54-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:03.756911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:03.756758 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lvd\" (UniqueName: \"kubernetes.io/projected/06a8c416-4abb-4cd5-9e8b-f30707700d54-kube-api-access-95lvd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:04.360437 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:04.360399 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" event={"ID":"06a8c416-4abb-4cd5-9e8b-f30707700d54","Type":"ContainerDied","Data":"385b7517dffb491d047d7fe7f7e5f08cec02a0201d432115834b381b2ebecca6"} Oct 29 23:18:04.360623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:04.360452 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="385b7517dffb491d047d7fe7f7e5f08cec02a0201d432115834b381b2ebecca6" Oct 29 23:18:04.360623 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:04.360415 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gsdmp" Oct 29 23:18:09.500065 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.500021 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" podUID="b7d37349-b6d7-4a79-9c9c-bee9ba40422b" containerName="main" containerID="cri-o://a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755" gracePeriod=45 Oct 29 23:18:09.759027 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.758969 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:18:09.805811 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.805780 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home\") pod \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " Oct 29 23:18:09.805987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.805864 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir\") pod \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " Oct 29 23:18:09.805987 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.805915 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp\" (UniqueName: \"kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp\") pod \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " Oct 29 23:18:09.806087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.805983 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache\") pod \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " Oct 29 23:18:09.806087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.806006 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home" pod "b7d37349-b6d7-4a79-9c9c-bee9ba40422b" (UID: "b7d37349-b6d7-4a79-9c9c-bee9ba40422b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:09.806087 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.806030 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm\") pod \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " Oct 29 23:18:09.806195 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.806103 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs\") pod \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\" (UID: \"b7d37349-b6d7-4a79-9c9c-bee9ba40422b\") " Oct 29 23:18:09.806815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.806414 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:09.806815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.806452 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir" pod "b7d37349-b6d7-4a79-9c9c-bee9ba40422b" (UID: "b7d37349-b6d7-4a79-9c9c-bee9ba40422b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:09.806815 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.806672 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache" pod "b7d37349-b6d7-4a79-9c9c-bee9ba40422b" (UID: "b7d37349-b6d7-4a79-9c9c-bee9ba40422b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:09.808755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.808678 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp" pod "b7d37349-b6d7-4a79-9c9c-bee9ba40422b" (UID: "b7d37349-b6d7-4a79-9c9c-bee9ba40422b"). InnerVolumeSpecName "kube-api-access-h7bsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:09.808977 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.808956 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs" pod "b7d37349-b6d7-4a79-9c9c-bee9ba40422b" (UID: "b7d37349-b6d7-4a79-9c9c-bee9ba40422b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:09.809165 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.809146 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm" pod "b7d37349-b6d7-4a79-9c9c-bee9ba40422b" (UID: "b7d37349-b6d7-4a79-9c9c-bee9ba40422b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:09.906896 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.906868 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:09.906896 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.906895 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:09.907063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.906903 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:09.907063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.906911 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:09.907063 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:09.906919 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7bsp\" (UniqueName: \"kubernetes.io/projected/b7d37349-b6d7-4a79-9c9c-bee9ba40422b-kube-api-access-h7bsp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:10.392833 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.392799 5367 generic.go:356] "Generic (PLEG): container finished" podID="b7d37349-b6d7-4a79-9c9c-bee9ba40422b" containerID="a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755" exitCode=0 Oct 29 23:18:10.393023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.392843 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" event={"ID":"b7d37349-b6d7-4a79-9c9c-bee9ba40422b","Type":"ContainerDied","Data":"a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755"} Oct 29 23:18:10.393023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.392854 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" Oct 29 23:18:10.393023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.392870 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98" event={"ID":"b7d37349-b6d7-4a79-9c9c-bee9ba40422b","Type":"ContainerDied","Data":"1707657a01ae0fb145362669cee0026539947ba4e326af54573d33e2324028d9"} Oct 29 23:18:10.393023 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.392887 5367 scope.go:122] "RemoveContainer" containerID="a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755" Oct 29 23:18:10.403138 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.403116 5367 scope.go:122] "RemoveContainer" containerID="a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755" Oct 29 23:18:10.403406 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:18:10.403382 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755\": container with ID starting with a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755 not found: ID does not exist" containerID="a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755" Oct 29 23:18:10.403459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.403416 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755"} err="failed to get container status \"a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755\": rpc error: code = NotFound desc = could not find container \"a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755\": container with ID starting with a70d38ceb2379c496b41de280b580577643c86e06df0897f0ecd7980901ec755 not found: ID does not exist" Oct 29 23:18:10.449829 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.449798 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98"] Oct 29 23:18:10.453181 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:10.453158 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e33378d4-kserve-859d798777-czq98"] Oct 29 23:18:11.219620 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:11.219589 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d37349-b6d7-4a79-9c9c-bee9ba40422b" path="/var/lib/kubelet/pods/b7d37349-b6d7-4a79-9c9c-bee9ba40422b/volumes" Oct 29 23:18:11.728912 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:11.728872 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" podUID="18904c37-7555-450b-b8f8-95e7bfb98aa7" containerName="main" containerID="cri-o://d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b" gracePeriod=45 Oct 29 23:18:12.246394 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.246364 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:18:12.338672 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.338621 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg\" (UniqueName: \"kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg\") pod \"18904c37-7555-450b-b8f8-95e7bfb98aa7\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " Oct 29 23:18:12.338907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.338681 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir\") pod \"18904c37-7555-450b-b8f8-95e7bfb98aa7\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " Oct 29 23:18:12.338907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.338740 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home\") pod \"18904c37-7555-450b-b8f8-95e7bfb98aa7\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " Oct 29 23:18:12.338907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.338817 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache\") pod \"18904c37-7555-450b-b8f8-95e7bfb98aa7\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " Oct 29 23:18:12.338907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.338887 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm\") pod \"18904c37-7555-450b-b8f8-95e7bfb98aa7\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " Oct 29 23:18:12.339159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.338954 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs\" (UniqueName: \"kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs\") pod \"18904c37-7555-450b-b8f8-95e7bfb98aa7\" (UID: \"18904c37-7555-450b-b8f8-95e7bfb98aa7\") " Oct 29 23:18:12.339159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.339005 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir" pod "18904c37-7555-450b-b8f8-95e7bfb98aa7" (UID: "18904c37-7555-450b-b8f8-95e7bfb98aa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:12.339159 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.339039 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home" pod "18904c37-7555-450b-b8f8-95e7bfb98aa7" (UID: "18904c37-7555-450b-b8f8-95e7bfb98aa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:12.339389 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.339166 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache" pod "18904c37-7555-450b-b8f8-95e7bfb98aa7" (UID: "18904c37-7555-450b-b8f8-95e7bfb98aa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:12.339634 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.339613 5367 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:12.339743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.339639 5367 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:12.339743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.339653 5367 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:12.341420 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.341401 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg" pod "18904c37-7555-450b-b8f8-95e7bfb98aa7" (UID: "18904c37-7555-450b-b8f8-95e7bfb98aa7"). InnerVolumeSpecName "kube-api-access-nddwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:12.341613 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.341588 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs" pod "18904c37-7555-450b-b8f8-95e7bfb98aa7" (UID: "18904c37-7555-450b-b8f8-95e7bfb98aa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:12.341613 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.341575 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm" pod "18904c37-7555-450b-b8f8-95e7bfb98aa7" (UID: "18904c37-7555-450b-b8f8-95e7bfb98aa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:12.427374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.427156 5367 generic.go:356] "Generic (PLEG): container finished" podID="18904c37-7555-450b-b8f8-95e7bfb98aa7" containerID="d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b" exitCode=0 Oct 29 23:18:12.427374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.427235 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" event={"ID":"18904c37-7555-450b-b8f8-95e7bfb98aa7","Type":"ContainerDied","Data":"d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b"} Oct 29 23:18:12.427374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.427291 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" event={"ID":"18904c37-7555-450b-b8f8-95e7bfb98aa7","Type":"ContainerDied","Data":"68fd3f19b9cc0a8ecf0a60601ec5af6489e4ecf492ac7184cffdc571220bdd1b"} Oct 29 23:18:12.427374 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.427315 5367 scope.go:122] "RemoveContainer" containerID="d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b" Oct 29 23:18:12.427829 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.427635 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" Oct 29 23:18:12.440232 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.440148 5367 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18904c37-7555-450b-b8f8-95e7bfb98aa7-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:12.440232 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.440166 5367 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18904c37-7555-450b-b8f8-95e7bfb98aa7-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:12.440232 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.440177 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddwg\" (UniqueName: \"kubernetes.io/projected/18904c37-7555-450b-b8f8-95e7bfb98aa7-kube-api-access-nddwg\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:12.440359 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.440262 5367 scope.go:122] "RemoveContainer" containerID="d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b" Oct 29 23:18:12.440527 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:18:12.440506 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b\": container with ID starting with d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b not found: ID does not exist" containerID="d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b" Oct 29 23:18:12.440578 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.440536 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b"} err="failed to get container status \"d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b\": rpc error: code = NotFound desc = could not find container \"d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b\": container with ID starting with d6269211dde6cc3ec056cf408bb420d4970b6c2aca7f638c5ac322ea50c3680b not found: ID does not exist" Oct 29 23:18:12.518768 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.517741 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q"] Oct 29 23:18:12.521284 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:12.521259 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q"] Oct 29 23:18:13.152677 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:13.152635 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-e33378d4-kserve-c658b6fb9-l8d6q" podUID="18904c37-7555-450b-b8f8-95e7bfb98aa7" containerName="main" probeResult="failure" output="Get \"https://10.133.1.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:18:13.231306 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:13.231271 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18904c37-7555-450b-b8f8-95e7bfb98aa7" path="/var/lib/kubelet/pods/18904c37-7555-450b-b8f8-95e7bfb98aa7/volumes" Oct 29 23:18:22.333512 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:18:22.333475 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:18:22.850012 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.849977 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md"] Oct 29 23:18:22.850368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850356 5367 cpu_manager.go:414] "containerMap: removing container" podUID="18904c37-7555-450b-b8f8-95e7bfb98aa7" containerName="main" Oct 29 23:18:22.850427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850369 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18904c37-7555-450b-b8f8-95e7bfb98aa7" containerName="main" Oct 29 23:18:22.850427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850399 5367 cpu_manager.go:414] "containerMap: removing container" podUID="b7d37349-b6d7-4a79-9c9c-bee9ba40422b" containerName="main" Oct 29 23:18:22.850427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850405 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d37349-b6d7-4a79-9c9c-bee9ba40422b" containerName="main" Oct 29 23:18:22.850427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850417 5367 cpu_manager.go:414] "containerMap: removing container" podUID="06a8c416-4abb-4cd5-9e8b-f30707700d54" containerName="revoke-keys" Oct 29 23:18:22.850427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850424 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a8c416-4abb-4cd5-9e8b-f30707700d54" containerName="revoke-keys" Oct 29 23:18:22.850591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850489 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="18904c37-7555-450b-b8f8-95e7bfb98aa7" containerName="main" Oct 29 23:18:22.850591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850501 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d37349-b6d7-4a79-9c9c-bee9ba40422b" containerName="main" Oct 29 23:18:22.850591 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.850507 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a8c416-4abb-4cd5-9e8b-f30707700d54" containerName="revoke-keys" Oct 29 23:18:22.854924 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.854907 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:22.875181 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.875158 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md"] Oct 29 23:18:22.886347 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.886325 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:18:22.886479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.886348 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:18:22.886479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.886396 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:18:22.886479 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.886440 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:18:22.962658 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.962631 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:22.962807 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.962761 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:22.962884 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:22.962864 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzn8f\" (UniqueName: \"kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.063509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.063481 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.063635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.063546 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzn8f\" (UniqueName: \"kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.063635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.063614 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.064288 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.064269 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.066109 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.066086 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.084921 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.084899 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzn8f\" (UniqueName: \"kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f\") pod \"maas-api-revoke-sub-a8deb641-mp9md\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.168423 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.168366 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:23.464851 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.464822 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md"] Oct 29 23:18:23.466510 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:18:23.466484 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40945653_df4e_4d6b_b754_83c5c2d9c6f3.slice/crio-8fbb550742e415bfc02d46d0910efa8e245c0c9e9256a46ef3afbf609890d079 WatchSource:0}: Error finding container 8fbb550742e415bfc02d46d0910efa8e245c0c9e9256a46ef3afbf609890d079: Status 404 returned error can't find the container with id 8fbb550742e415bfc02d46d0910efa8e245c0c9e9256a46ef3afbf609890d079 Oct 29 23:18:23.507150 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:23.507122 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" event={"ID":"40945653-df4e-4d6b-b754-83c5c2d9c6f3","Type":"ContainerStarted","Data":"8fbb550742e415bfc02d46d0910efa8e245c0c9e9256a46ef3afbf609890d079"} Oct 29 23:18:24.514412 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:24.514368 5367 generic.go:356] "Generic (PLEG): container finished" podID="40945653-df4e-4d6b-b754-83c5c2d9c6f3" containerID="01c70ed72bdadcba6c1deb21939b1fcf455aa195d17e36adf4d7896e4173c3b1" exitCode=0 Oct 29 23:18:24.514819 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:24.514413 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" event={"ID":"40945653-df4e-4d6b-b754-83c5c2d9c6f3","Type":"ContainerDied","Data":"01c70ed72bdadcba6c1deb21939b1fcf455aa195d17e36adf4d7896e4173c3b1"} Oct 29 23:18:25.647312 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.647288 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:25.687778 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.687755 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca\") pod \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " Oct 29 23:18:25.687904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.687790 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca\") pod \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " Oct 29 23:18:25.687904 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.687847 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f\" (UniqueName: \"kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f\") pod \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\" (UID: \"40945653-df4e-4d6b-b754-83c5c2d9c6f3\") " Oct 29 23:18:25.688158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.688121 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca" pod "40945653-df4e-4d6b-b754-83c5c2d9c6f3" (UID: "40945653-df4e-4d6b-b754-83c5c2d9c6f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:25.689895 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.689864 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca" pod "40945653-df4e-4d6b-b754-83c5c2d9c6f3" (UID: "40945653-df4e-4d6b-b754-83c5c2d9c6f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:25.690017 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.689920 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f" pod "40945653-df4e-4d6b-b754-83c5c2d9c6f3" (UID: "40945653-df4e-4d6b-b754-83c5c2d9c6f3"). InnerVolumeSpecName "kube-api-access-qzn8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:25.789535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.789466 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-openshift-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:25.789535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.789492 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40945653-df4e-4d6b-b754-83c5c2d9c6f3-maas-api-xks-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:25.789535 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:25.789504 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzn8f\" (UniqueName: \"kubernetes.io/projected/40945653-df4e-4d6b-b754-83c5c2d9c6f3-kube-api-access-qzn8f\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:26.526346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:26.526313 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" event={"ID":"40945653-df4e-4d6b-b754-83c5c2d9c6f3","Type":"ContainerDied","Data":"8fbb550742e415bfc02d46d0910efa8e245c0c9e9256a46ef3afbf609890d079"} Oct 29 23:18:26.526346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:26.526340 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8deb641-mp9md" Oct 29 23:18:26.526572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:26.526361 5367 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fbb550742e415bfc02d46d0910efa8e245c0c9e9256a46ef3afbf609890d079" Oct 29 23:18:35.255572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.255538 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx"] Oct 29 23:18:35.256007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.255911 5367 cpu_manager.go:414] "containerMap: removing container" podUID="40945653-df4e-4d6b-b754-83c5c2d9c6f3" containerName="revoke-keys" Oct 29 23:18:35.256007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.255923 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40945653-df4e-4d6b-b754-83c5c2d9c6f3" containerName="revoke-keys" Oct 29 23:18:35.256007 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.255988 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="40945653-df4e-4d6b-b754-83c5c2d9c6f3" containerName="revoke-keys" Oct 29 23:18:35.259162 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.259140 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.272434 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.272404 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-jf5k8"] Oct 29 23:18:35.275742 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.275725 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.284197 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.284173 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:18:35.284365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.284204 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:18:35.284365 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.284275 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 23:18:35.292144 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.292119 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx"] Oct 29 23:18:35.297475 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.297441 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-jf5k8"] Oct 29 23:18:35.376855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.376829 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 29 23:18:35.376855 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.376831 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 23:18:35.380322 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380300 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.380400 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380336 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-maas-api-tls\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.380439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380422 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-openshift-service-ca\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.380476 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380450 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.380549 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380537 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpq5\" (UniqueName: \"kubernetes.io/projected/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-kube-api-access-fwpq5\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.380594 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380557 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5l6\" (UniqueName: \"kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.380631 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380591 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-metrics-certs\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.380667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.380629 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.481691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481663 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpq5\" (UniqueName: \"kubernetes.io/projected/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-kube-api-access-fwpq5\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.481691 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481692 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5l6\" (UniqueName: \"kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.481920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481752 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-metrics-certs\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.481920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481770 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.481920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481791 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.481920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481816 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-maas-api-tls\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.481920 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481894 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-openshift-service-ca\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.482129 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.481942 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.482740 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.482713 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-openshift-service-ca\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.482894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.482862 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.484439 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.484410 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-metrics-certs\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.484550 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.484450 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.484550 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.484451 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.484629 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.484569 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-maas-api-tls\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.503472 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.503446 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpq5\" (UniqueName: \"kubernetes.io/projected/b373e1ff-d5d7-4417-bc51-269a05e2ff2f-kube-api-access-fwpq5\") pod \"maas-api-595cc648c-jf5k8\" (UID: \"b373e1ff-d5d7-4417-bc51-269a05e2ff2f\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.503572 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.503549 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5l6\" (UniqueName: \"kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6\") pod \"maas-api-5c48877c75-xm9wx\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.572223 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.572190 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:35.597213 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.597186 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:35.889635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.889607 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx"] Oct 29 23:18:35.891189 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:18:35.891163 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8076144e_eff5_42fb_b7dd_cc371bdc575a.slice/crio-5402206b0259f803490f89f60587f3e1255760728e9be11894700624d048f3b5 WatchSource:0}: Error finding container 5402206b0259f803490f89f60587f3e1255760728e9be11894700624d048f3b5: Status 404 returned error can't find the container with id 5402206b0259f803490f89f60587f3e1255760728e9be11894700624d048f3b5 Oct 29 23:18:35.892373 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.892357 5367 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:18:35.925589 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:35.925562 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-jf5k8"] Oct 29 23:18:35.927623 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:18:35.927596 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb373e1ff_d5d7_4417_bc51_269a05e2ff2f.slice/crio-7fbb82b7c239655ba723254b0b30619ac1a7fb0cae97233f63669626eee3a4de WatchSource:0}: Error finding container 7fbb82b7c239655ba723254b0b30619ac1a7fb0cae97233f63669626eee3a4de: Status 404 returned error can't find the container with id 7fbb82b7c239655ba723254b0b30619ac1a7fb0cae97233f63669626eee3a4de Oct 29 23:18:36.588459 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.588422 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" event={"ID":"b373e1ff-d5d7-4417-bc51-269a05e2ff2f","Type":"ContainerStarted","Data":"d6c8bd87ebbb88349eee177f7fbc79071b650d05a050f662f4db6f5327e77ab0"} Oct 29 23:18:36.588964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.588482 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" event={"ID":"b373e1ff-d5d7-4417-bc51-269a05e2ff2f","Type":"ContainerStarted","Data":"7fbb82b7c239655ba723254b0b30619ac1a7fb0cae97233f63669626eee3a4de"} Oct 29 23:18:36.588964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.588608 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:36.590323 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.590294 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" event={"ID":"8076144e-eff5-42fb-b7dd-cc371bdc575a","Type":"ContainerStarted","Data":"e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de"} Oct 29 23:18:36.590451 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.590345 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" event={"ID":"8076144e-eff5-42fb-b7dd-cc371bdc575a","Type":"ContainerStarted","Data":"5402206b0259f803490f89f60587f3e1255760728e9be11894700624d048f3b5"} Oct 29 23:18:36.590451 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.590430 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:36.635779 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.635731 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" podStartSLOduration=1.2712587690000001 podStartE2EDuration="1.635716745s" totalImagesPullingTime="364.457976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:36.632668242 +0000 UTC m=+3783.526743159" watchObservedRunningTime="2026-10-29 23:18:36.635716745 +0000 UTC m=+3783.529791663" Oct 29 23:18:36.678828 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:36.678767 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" podStartSLOduration=1.298666068 podStartE2EDuration="1.678753551s" totalImagesPullingTime="380.087483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:36.676398725 +0000 UTC m=+3783.570473647" watchObservedRunningTime="2026-10-29 23:18:36.678753551 +0000 UTC m=+3783.572828471" Oct 29 23:18:37.028444 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:37.028359 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6"] Oct 29 23:18:37.030490 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:37.030459 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncfl6"] Oct 29 23:18:37.220123 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:37.220092 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755a9360-3650-4b4a-8317-6f12133b664a" path="/var/lib/kubelet/pods/755a9360-3650-4b4a-8317-6f12133b664a/volumes" Oct 29 23:18:40.527710 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:40.527676 5367 scope.go:122] "RemoveContainer" containerID="1b462758c3f58326ca7410db7861f6f6493807e17a658dd3a83c72396e9ec9f9" Oct 29 23:18:40.540033 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:40.539920 5367 scope.go:122] "RemoveContainer" containerID="2558a49a7c037cebf1355a5ef3bdd495e1d807ead090a0e362a47449ff001f5b" Oct 29 23:18:42.601792 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:42.601763 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-jf5k8" Oct 29 23:18:42.602368 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:42.602281 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:42.678944 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:42.678896 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx"] Oct 29 23:18:42.679229 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:42.679191 5367 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" podUID="8076144e-eff5-42fb-b7dd-cc371bdc575a" containerName="maas-api" containerID="cri-o://e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de" gracePeriod=30 Oct 29 23:18:42.927918 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:42.927891 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:43.057962 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.057927 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6\" (UniqueName: \"kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6\") pod \"8076144e-eff5-42fb-b7dd-cc371bdc575a\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " Oct 29 23:18:43.058158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.057996 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca\") pod \"8076144e-eff5-42fb-b7dd-cc371bdc575a\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " Oct 29 23:18:43.058158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.058051 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs\") pod \"8076144e-eff5-42fb-b7dd-cc371bdc575a\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " Oct 29 23:18:43.058158 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.058092 5367 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls\") pod \"8076144e-eff5-42fb-b7dd-cc371bdc575a\" (UID: \"8076144e-eff5-42fb-b7dd-cc371bdc575a\") " Oct 29 23:18:43.058438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.058411 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca" pod "8076144e-eff5-42fb-b7dd-cc371bdc575a" (UID: "8076144e-eff5-42fb-b7dd-cc371bdc575a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:43.060108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.060085 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls" pod "8076144e-eff5-42fb-b7dd-cc371bdc575a" (UID: "8076144e-eff5-42fb-b7dd-cc371bdc575a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:43.060214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.060182 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs" pod "8076144e-eff5-42fb-b7dd-cc371bdc575a" (UID: "8076144e-eff5-42fb-b7dd-cc371bdc575a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:43.060308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.060285 5367 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6" pod "8076144e-eff5-42fb-b7dd-cc371bdc575a" (UID: "8076144e-eff5-42fb-b7dd-cc371bdc575a"). InnerVolumeSpecName "kube-api-access-lw5l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:43.159075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.158979 5367 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw5l6\" (UniqueName: \"kubernetes.io/projected/8076144e-eff5-42fb-b7dd-cc371bdc575a-kube-api-access-lw5l6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.159075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.159011 5367 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076144e-eff5-42fb-b7dd-cc371bdc575a-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.159075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.159022 5367 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-metrics-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.159075 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.159031 5367 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8076144e-eff5-42fb-b7dd-cc371bdc575a-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.629050 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.629014 5367 generic.go:356] "Generic (PLEG): container finished" podID="8076144e-eff5-42fb-b7dd-cc371bdc575a" containerID="e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de" exitCode=0 Oct 29 23:18:43.629520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.629057 5367 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" Oct 29 23:18:43.629520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.629094 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" event={"ID":"8076144e-eff5-42fb-b7dd-cc371bdc575a","Type":"ContainerDied","Data":"e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de"} Oct 29 23:18:43.629520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.629157 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx" event={"ID":"8076144e-eff5-42fb-b7dd-cc371bdc575a","Type":"ContainerDied","Data":"5402206b0259f803490f89f60587f3e1255760728e9be11894700624d048f3b5"} Oct 29 23:18:43.629520 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.629179 5367 scope.go:122] "RemoveContainer" containerID="e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de" Oct 29 23:18:43.638434 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.638394 5367 scope.go:122] "RemoveContainer" containerID="e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de" Oct 29 23:18:43.638692 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:18:43.638675 5367 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de\": container with ID starting with e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de not found: ID does not exist" containerID="e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de" Oct 29 23:18:43.638766 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.638723 5367 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de"} err="failed to get container status \"e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de\": rpc error: code = NotFound desc = could not find container \"e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de\": container with ID starting with e53dd184f5153663dbcd604429d683e4a4ba3f1c203d72d20421ec2eb20299de not found: ID does not exist" Oct 29 23:18:43.684964 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.684929 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx"] Oct 29 23:18:43.698911 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:43.698881 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xm9wx"] Oct 29 23:18:45.231385 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:45.231156 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8076144e-eff5-42fb-b7dd-cc371bdc575a" path="/var/lib/kubelet/pods/8076144e-eff5-42fb-b7dd-cc371bdc575a/volumes" Oct 29 23:18:46.357101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:46.355882 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:18:46.357101 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:46.355945 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kthhw" podUID="d70fe24b-0114-4d0a-9739-e057bac7f836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:18:52.330884 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:18:52.330842 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:18:54.826979 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:54.826904 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:18:54.827661 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:18:54.826995 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:19:09.828870 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:09.828832 5367 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:19:09.829488 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:09.829455 5367 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn" podUID="ee2a1d1e-92f2-4ad9-bcff-d180db897e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:19:22.330140 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:19:22.330098 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:19:33.328240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:33.328211 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97d6577-xkcfk_611852ef-ca5e-4e8f-b58b-277efda1708c/authorino/0.log" Oct 29 23:19:40.572360 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:40.572329 5367 scope.go:122] "RemoveContainer" containerID="f292440bc47cb406749993b25719cd92309d9371edf3372ec2951ad30414a904" Oct 29 23:19:43.957218 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:43.957186 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-85bb745f44-xt49q_87fc931d-3b42-4162-ba49-18e5a06dcb35/manager/0.log" Oct 29 23:19:46.283398 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:46.283367 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-cv5kd_7d8ff79f-75a0-4983-a84c-2dbdb32e255b/copy-manifests/0.log" Oct 29 23:19:46.310635 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:46.310602 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-cv5kd_7d8ff79f-75a0-4983-a84c-2dbdb32e255b/manager/0.log" Oct 29 23:19:47.826041 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:47.826007 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-jf5k8_b373e1ff-d5d7-4417-bc51-269a05e2ff2f/maas-api/0.log" Oct 29 23:19:48.400716 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:48.400680 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gsdmp_06a8c416-4abb-4cd5-9e8b-f30707700d54/revoke-keys/0.log" Oct 29 23:19:48.583199 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:48.583168 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-l9gf2_991f4492-f12a-4b6f-9a9e-d8b4321809e2/revoke-keys/0.log" Oct 29 23:19:48.951446 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:48.951412 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a8deb641-mp9md_40945653-df4e-4d6b-b754-83c5c2d9c6f3/revoke-keys/0.log" Oct 29 23:19:49.151427 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:49.151401 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-7ljhx_8c787cee-779c-4070-bc24-260bd7af1f0a/revoke-keys/0.log" Oct 29 23:19:49.338058 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:49.338030 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-dgsdg_f012b6fd-3622-4fc4-ae74-55a3e05abd14/revoke-keys/0.log" Oct 29 23:19:51.545226 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:51.545188 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97d6577-xkcfk_611852ef-ca5e-4e8f-b58b-277efda1708c/authorino/0.log" Oct 29 23:19:52.186291 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:52.186263 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-2vqxx_e03eaa49-3506-46db-ae0f-6df0e4973111/kuadrant-console-plugin/0.log" Oct 29 23:19:52.330318 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:19:52.330284 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:19:54.252542 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:54.252511 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b649d648f-s7zqw_d46945c5-160d-485a-a63a-e90f7e2be9fc/kube-auth-proxy/0.log" Oct 29 23:19:54.660211 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:54.660177 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx_2bcf97eb-0392-4819-8114-e0c06d076ca6/istio-proxy/0.log" Oct 29 23:19:55.275079 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:55.275049 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-kthhw_d70fe24b-0114-4d0a-9739-e057bac7f836/istio-proxy/0.log" Oct 29 23:19:56.266689 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:56.266659 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn_ee2a1d1e-92f2-4ad9-bcff-d180db897e67/istio-proxy/0.log" Oct 29 23:19:56.478897 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:56.478864 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8q5x4_ae46657d-fb4f-4c83-8c75-b38523884064/payload-pre-processing/0.log" Oct 29 23:19:57.629225 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:57.629191 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bx292_88e74199-7eba-4437-b7e7-d2dc4094d87e/main/0.log" Oct 29 23:19:57.820636 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:57.820602 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zz4np_d4dfa063-b32f-46d7-975a-378ad1d59be9/main/0.log" Oct 29 23:19:58.016419 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:58.016332 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zzjpn_8f18feb0-835d-45e8-a8e2-0053321ce57a/main/0.log" Oct 29 23:19:58.769780 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:58.769742 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb_ba4c4883-c331-458f-8060-c35e65211fd4/main/0.log" Oct 29 23:19:59.678646 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:19:59.678616 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97d6577-xkcfk_611852ef-ca5e-4e8f-b58b-277efda1708c/authorino/0.log" Oct 29 23:20:08.780990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:08.780953 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-85bb745f44-xt49q_87fc931d-3b42-4162-ba49-18e5a06dcb35/manager/0.log" Oct 29 23:20:10.551797 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:10.551769 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-cv5kd_7d8ff79f-75a0-4983-a84c-2dbdb32e255b/copy-manifests/0.log" Oct 29 23:20:10.582737 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:10.582710 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-cv5kd_7d8ff79f-75a0-4983-a84c-2dbdb32e255b/manager/0.log" Oct 29 23:20:11.834881 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:11.834848 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-jf5k8_b373e1ff-d5d7-4417-bc51-269a05e2ff2f/maas-api/0.log" Oct 29 23:20:12.414813 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:12.414778 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gsdmp_06a8c416-4abb-4cd5-9e8b-f30707700d54/revoke-keys/0.log" Oct 29 23:20:12.602894 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:12.602859 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-l9gf2_991f4492-f12a-4b6f-9a9e-d8b4321809e2/revoke-keys/0.log" Oct 29 23:20:12.964910 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:12.964874 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a8deb641-mp9md_40945653-df4e-4d6b-b754-83c5c2d9c6f3/revoke-keys/0.log" Oct 29 23:20:13.160135 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:13.160103 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-7ljhx_8c787cee-779c-4070-bc24-260bd7af1f0a/revoke-keys/0.log" Oct 29 23:20:13.353271 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:13.353241 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-dgsdg_f012b6fd-3622-4fc4-ae74-55a3e05abd14/revoke-keys/0.log" Oct 29 23:20:15.061762 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:15.061663 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97d6577-xkcfk_611852ef-ca5e-4e8f-b58b-277efda1708c/authorino/0.log" Oct 29 23:20:15.676657 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:15.676628 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-2vqxx_e03eaa49-3506-46db-ae0f-6df0e4973111/kuadrant-console-plugin/0.log" Oct 29 23:20:17.441547 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:17.441517 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b649d648f-s7zqw_d46945c5-160d-485a-a63a-e90f7e2be9fc/kube-auth-proxy/0.log" Oct 29 23:20:17.831408 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:17.831381 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx_2bcf97eb-0392-4819-8114-e0c06d076ca6/istio-proxy/0.log" Oct 29 23:20:18.423069 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:18.423039 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-kthhw_d70fe24b-0114-4d0a-9739-e057bac7f836/istio-proxy/0.log" Oct 29 23:20:19.420329 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:19.420296 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn_ee2a1d1e-92f2-4ad9-bcff-d180db897e67/istio-proxy/0.log" Oct 29 23:20:19.626827 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:19.626800 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8q5x4_ae46657d-fb4f-4c83-8c75-b38523884064/payload-pre-processing/0.log" Oct 29 23:20:20.567390 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:20.567358 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bx292_88e74199-7eba-4437-b7e7-d2dc4094d87e/main/0.log" Oct 29 23:20:20.760686 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:20.760657 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zz4np_d4dfa063-b32f-46d7-975a-378ad1d59be9/main/0.log" Oct 29 23:20:20.952940 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:20.952864 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zzjpn_8f18feb0-835d-45e8-a8e2-0053321ce57a/main/0.log" Oct 29 23:20:21.723531 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:21.723499 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2cbqb_ba4c4883-c331-458f-8060-c35e65211fd4/main/0.log" Oct 29 23:20:22.329811 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:20:22.329774 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:20:23.027045 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:23.027013 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg"] Oct 29 23:20:23.040549 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:23.040519 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dgsdg"] Oct 29 23:20:23.219610 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:23.219580 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f012b6fd-3622-4fc4-ae74-55a3e05abd14" path="/var/lib/kubelet/pods/f012b6fd-3622-4fc4-ae74-55a3e05abd14/volumes" Oct 29 23:20:40.033005 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:40.032971 5367 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx"] Oct 29 23:20:40.046076 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:40.046038 5367 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7ljhx"] Oct 29 23:20:41.219994 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:20:41.219965 5367 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c787cee-779c-4070-bc24-260bd7af1f0a" path="/var/lib/kubelet/pods/8c787cee-779c-4070-bc24-260bd7af1f0a/volumes" Oct 29 23:20:52.330096 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:20:52.330057 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:21:22.330074 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:21:22.330038 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:21:40.604791 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:40.604758 5367 scope.go:122] "RemoveContainer" containerID="199797c0328e84ca003feb530a1e6c651d255e6ed48b279a7f8d693ef12eeb0a" Oct 29 23:21:40.613767 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:40.613742 5367 scope.go:122] "RemoveContainer" containerID="b43c62a5c4aaac7a5b122fc8590d32f96b38e1eac208ba813e5ff948507a1f34" Oct 29 23:21:41.502574 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.502537 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4b5wq/must-gather-fkcd6"] Oct 29 23:21:41.502935 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.502921 5367 cpu_manager.go:414] "containerMap: removing container" podUID="8076144e-eff5-42fb-b7dd-cc371bdc575a" containerName="maas-api" Oct 29 23:21:41.502990 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.502936 5367 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8076144e-eff5-42fb-b7dd-cc371bdc575a" containerName="maas-api" Oct 29 23:21:41.503029 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.503015 5367 memory_manager.go:365] "RemoveStaleState removing state" podUID="8076144e-eff5-42fb-b7dd-cc371bdc575a" containerName="maas-api" Oct 29 23:21:41.507682 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.507663 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.523443 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.523421 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b5wq\"/\"kube-root-ca.crt\"" Oct 29 23:21:41.536671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.536648 5367 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4b5wq\"/\"default-dockercfg-7vvsf\"" Oct 29 23:21:41.536671 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.536670 5367 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b5wq\"/\"openshift-service-ca.crt\"" Oct 29 23:21:41.539346 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.539324 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/must-gather-fkcd6"] Oct 29 23:21:41.546200 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.546174 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7fx\" (UniqueName: \"kubernetes.io/projected/239f3443-2132-4221-9ecc-776d55385c64-kube-api-access-ps7fx\") pod \"must-gather-fkcd6\" (UID: \"239f3443-2132-4221-9ecc-776d55385c64\") " pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.546324 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.546247 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/239f3443-2132-4221-9ecc-776d55385c64-must-gather-output\") pod \"must-gather-fkcd6\" (UID: \"239f3443-2132-4221-9ecc-776d55385c64\") " pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.647093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.647055 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/239f3443-2132-4221-9ecc-776d55385c64-must-gather-output\") pod \"must-gather-fkcd6\" (UID: \"239f3443-2132-4221-9ecc-776d55385c64\") " pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.647509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.647168 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7fx\" (UniqueName: \"kubernetes.io/projected/239f3443-2132-4221-9ecc-776d55385c64-kube-api-access-ps7fx\") pod \"must-gather-fkcd6\" (UID: \"239f3443-2132-4221-9ecc-776d55385c64\") " pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.647509 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.647402 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/239f3443-2132-4221-9ecc-776d55385c64-must-gather-output\") pod \"must-gather-fkcd6\" (UID: \"239f3443-2132-4221-9ecc-776d55385c64\") " pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.671755 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.671694 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7fx\" (UniqueName: \"kubernetes.io/projected/239f3443-2132-4221-9ecc-776d55385c64-kube-api-access-ps7fx\") pod \"must-gather-fkcd6\" (UID: \"239f3443-2132-4221-9ecc-776d55385c64\") " pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:41.821615 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:41.821587 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/must-gather-fkcd6" Oct 29 23:21:42.117272 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:42.117246 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/must-gather-fkcd6"] Oct 29 23:21:42.119288 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:21:42.119255 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod239f3443_2132_4221_9ecc_776d55385c64.slice/crio-89f6c57be6bd39a692df1cf30537b6e7e2dffe863495b7e3ce0e03ae3f0cb1e2 WatchSource:0}: Error finding container 89f6c57be6bd39a692df1cf30537b6e7e2dffe863495b7e3ce0e03ae3f0cb1e2: Status 404 returned error can't find the container with id 89f6c57be6bd39a692df1cf30537b6e7e2dffe863495b7e3ce0e03ae3f0cb1e2 Oct 29 23:21:42.621397 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:42.621354 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/must-gather-fkcd6" event={"ID":"239f3443-2132-4221-9ecc-776d55385c64","Type":"ContainerStarted","Data":"89f6c57be6bd39a692df1cf30537b6e7e2dffe863495b7e3ce0e03ae3f0cb1e2"} Oct 29 23:21:43.627960 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:43.627923 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/must-gather-fkcd6" event={"ID":"239f3443-2132-4221-9ecc-776d55385c64","Type":"ContainerStarted","Data":"fcbd9ee0925faa4132f29fdec0a48ec1aee974da1fbc735db937ad280e17dd28"} Oct 29 23:21:43.628376 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:43.627970 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/must-gather-fkcd6" event={"ID":"239f3443-2132-4221-9ecc-776d55385c64","Type":"ContainerStarted","Data":"3cf71ac7dc6673a9e2d8b6f786b148d1eb9331b77aabb89a787a0ef2da6e2523"} Oct 29 23:21:43.667743 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:43.667658 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4b5wq/must-gather-fkcd6" podStartSLOduration=1.9512469129999999 podStartE2EDuration="2.667642993s" totalImagesPullingTime="716.39608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:43.666743824 +0000 UTC m=+3970.560818745" watchObservedRunningTime="2026-10-29 23:21:43.667642993 +0000 UTC m=+3970.561717914" Oct 29 23:21:46.325955 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:46.325878 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rk9gl_2768c741-ba6f-4f8f-b20a-ac431cd8b83f/global-pull-secret-syncer/0.log" Oct 29 23:21:47.092132 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:47.090974 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cdbjg_192ace5d-ce40-41f2-b494-4dd3bfae0507/konnectivity-agent/0.log" Oct 29 23:21:47.570722 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:47.568625 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-29.ec2.internal_4ab51ab2ff14d7b6654efe4a89f4f347/haproxy/0.log" Oct 29 23:21:52.355759 ip-10-0-139-29 kubenswrapper[5367]: E1029 23:21:52.355681 5367 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n69kw" containerName="extractor" Oct 29 23:21:54.233723 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:54.233352 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97d6577-xkcfk_611852ef-ca5e-4e8f-b58b-277efda1708c/authorino/0.log" Oct 29 23:21:54.600966 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:54.600932 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-2vqxx_e03eaa49-3506-46db-ae0f-6df0e4973111/kuadrant-console-plugin/0.log" Oct 29 23:21:56.261580 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.261540 5367 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc"] Oct 29 23:21:56.274231 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.274203 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.300716 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.295840 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc"] Oct 29 23:21:56.396308 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.396246 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-proc\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.396517 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.396385 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-sys\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.396597 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.396507 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfsj\" (UniqueName: \"kubernetes.io/projected/54b56777-c526-4dfc-bf79-1005cf109ec7-kube-api-access-plfsj\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.396667 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.396594 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-lib-modules\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.396757 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.396672 5367 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-podres\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.412915 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.412891 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/alertmanager/0.log" Oct 29 23:21:56.501549 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.501506 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-sys\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.501871 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.501850 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfsj\" (UniqueName: \"kubernetes.io/projected/54b56777-c526-4dfc-bf79-1005cf109ec7-kube-api-access-plfsj\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.502043 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.502028 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-lib-modules\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.502161 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.502149 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-podres\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.502285 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.502273 5367 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-proc\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.502494 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.502480 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-proc\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.502641 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.502628 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-sys\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.503106 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.503085 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-lib-modules\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.503281 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.503266 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54b56777-c526-4dfc-bf79-1005cf109ec7-podres\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.524664 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.524587 5367 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfsj\" (UniqueName: \"kubernetes.io/projected/54b56777-c526-4dfc-bf79-1005cf109ec7-kube-api-access-plfsj\") pod \"perf-node-gather-daemonset-k88rc\" (UID: \"54b56777-c526-4dfc-bf79-1005cf109ec7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.534097 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.534068 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/config-reloader/0.log" Oct 29 23:21:56.605728 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.605671 5367 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:56.659217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.659177 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/kube-rbac-proxy-web/0.log" Oct 29 23:21:56.776093 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.776017 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/kube-rbac-proxy/0.log" Oct 29 23:21:56.896811 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.896747 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/kube-rbac-proxy-metric/0.log" Oct 29 23:21:56.921027 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:56.921003 5367 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc"] Oct 29 23:21:56.923065 ip-10-0-139-29 kubenswrapper[5367]: W1029 23:21:56.923023 5367 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod54b56777_c526_4dfc_bf79_1005cf109ec7.slice/crio-b3c171dcb08c9e5cb10eb2cd51f1ab05a1f03f4131f5f3a915d3a40c4912890d WatchSource:0}: Error finding container b3c171dcb08c9e5cb10eb2cd51f1ab05a1f03f4131f5f3a915d3a40c4912890d: Status 404 returned error can't find the container with id b3c171dcb08c9e5cb10eb2cd51f1ab05a1f03f4131f5f3a915d3a40c4912890d Oct 29 23:21:57.013181 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.013142 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/prom-label-proxy/0.log" Oct 29 23:21:57.134524 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.134493 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_592888da-77f8-4787-ad63-8d4a1a2a4b31/init-config-reloader/0.log" Oct 29 23:21:57.744575 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.744545 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-684967c95f-cf5zb_29897fb3-27ec-4f39-8f74-f96c5cba04a1/metrics-server/0.log" Oct 29 23:21:57.785505 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.785467 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" event={"ID":"54b56777-c526-4dfc-bf79-1005cf109ec7","Type":"ContainerStarted","Data":"60291f7a472e99bab8c581c0bb4cdb0a9462527173e6fa68111edaefb822489a"} Oct 29 23:21:57.785725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.785531 5367 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" event={"ID":"54b56777-c526-4dfc-bf79-1005cf109ec7","Type":"ContainerStarted","Data":"b3c171dcb08c9e5cb10eb2cd51f1ab05a1f03f4131f5f3a915d3a40c4912890d"} Oct 29 23:21:57.785725 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.785677 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:21:57.825907 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:57.825855 5367 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" podStartSLOduration=1.825841053 podStartE2EDuration="1.825841053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:57.824328247 +0000 UTC m=+3984.718403167" watchObservedRunningTime="2026-10-29 23:21:57.825841053 +0000 UTC m=+3984.719915972" Oct 29 23:21:58.688108 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:58.688075 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5ml4_7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/node-exporter/0.log" Oct 29 23:21:58.802217 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:58.802185 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5ml4_7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/kube-rbac-proxy/0.log" Oct 29 23:21:58.918214 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:21:58.918184 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5ml4_7bcdc9c9-6f01-44e9-a874-bf914bdd5c9f/init-textfile/0.log" Oct 29 23:22:00.564166 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:00.564129 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-767dbcc698-ncbkm_23715cbc-f9cd-496d-9e45-c2306e7aa75c/telemeter-client/0.log" Oct 29 23:22:00.679756 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:00.679725 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-767dbcc698-ncbkm_23715cbc-f9cd-496d-9e45-c2306e7aa75c/reload/0.log" Oct 29 23:22:00.794157 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:00.794127 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-767dbcc698-ncbkm_23715cbc-f9cd-496d-9e45-c2306e7aa75c/kube-rbac-proxy/0.log" Oct 29 23:22:03.806875 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:03.806847 5367 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-k88rc" Oct 29 23:22:04.943773 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:04.943745 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55dffb84cd-26pgk_fdb0aae7-06f9-4d61-8f4a-a46ce846917b/console/0.log" Oct 29 23:22:08.876240 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:08.876210 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7xrmb_71169ebf-0176-4dec-8ac5-8a5adf441324/dns/0.log" Oct 29 23:22:08.990420 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:08.990381 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7xrmb_71169ebf-0176-4dec-8ac5-8a5adf441324/kube-rbac-proxy/0.log" Oct 29 23:22:09.347292 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:09.347261 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zr4d_3189ad37-09b3-47dc-aa9e-e384d7a0ff21/dns-node-resolver/0.log" Oct 29 23:22:10.935962 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:10.935930 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mhvkt_425b6b6b-e38b-4ceb-b473-f1c397eba5d0/node-ca/0.log" Oct 29 23:22:11.334438 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:11.334397 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b649d648f-s7zqw_d46945c5-160d-485a-a63a-e90f7e2be9fc/kube-auth-proxy/0.log" Oct 29 23:22:11.458284 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:11.458252 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-bsvzx_2bcf97eb-0392-4819-8114-e0c06d076ca6/istio-proxy/0.log" Oct 29 23:22:11.638985 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:11.638903 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-kthhw_d70fe24b-0114-4d0a-9739-e057bac7f836/istio-proxy/0.log" Oct 29 23:22:11.941501 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:11.941425 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-vsbwn_ee2a1d1e-92f2-4ad9-bcff-d180db897e67/istio-proxy/0.log" Oct 29 23:22:11.999140 ip-10-0-139-29 kubenswrapper[5367]: I1029 23:22:11.999112 5367 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8q5x4_ae46657d-fb4f-4c83-8c75-b38523884064/payload-pre-processing/0.log"